Regression in xen-netfront on v3.6.

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

So I hadn't done a git bisection yet. But if I choose git commit:
4b24ff71108164e047cf2c95990b77651163e315
    Merge tag 'for-v3.6' of git://git.infradead.org/battery-2.6

    Pull battery updates from Anton Vorontsov:


everything works nicely. Anything past that, so these merges:

konrad@phenom:~/ssd/linux$ git log --oneline --merges 4b24ff71108164e047cf2c95990b77651163e315..linus/master
2d53492 Merge tag 'irqdomain-for-linus' of git://git.secretlab.ca/git/linux-2.6
ac694db Merge branch 'akpm' (Andrew's patch-bomb)
a40a1d3 Merge tag 'vfio-for-v3.6' of git://github.com/awilliam/linux-vfio
3e9a970 Merge tag 'random_for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/random
941c872 Merge tag 'rdma-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband
8762541 Merge branch 'v4l_for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-media
6dbb35b Merge tag 'nfs-for-3.6-2' of git://git.linux-nfs.org/projects/trondmy/linux-nfs
fd37ce3 Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
1da9b6b Merge branches 'cma', 'ipoib', 'ocrdma' and 'qib' into for-next
6aeea3e Merge remote-tracking branch 'origin' into irqdomain/next
931efdf Merge branch 'v4l_for_linus' into staging/for_v3.6
80c1834 Merge tag 'v3.5-rc6' into irqdomain/next

are the culprit. I think it might be the networking pull but not sure. Ian
any thoughts?

Using config file "/test.xm".
Started domain latest (id=2)
[    0.000000] console [hvc0] enabled, bootconsole disabled
[    0.000000] Xen: using vcpuop timer interface
[    0.000000] installing Xen timer for CPU 0
[    0.000000] tsc: Detected 2294.530 MHz processor
[    0.000999] Calibrating delay loop (skipped), value calculated using timer frequency.. 4589.06 BogoMIPS (lpj=2294530)
[    0.000999] pid_max: default: 32768 minimum: 301
[    0.000999] Security Framework initialized
[    0.000999] SELinux:  Initializing.
[    0.000999] SELinux:  Starting in permissive mode
[    0.000999] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
[    0.001520] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
[    0.001875] Mount-cache hash table entries: 256
[    0.002007] Initializing cgroup subsys cpuacct
[    0.002013] Initializing cgroup subsys freezer
[    0.002070] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
[    0.002070] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8)
[    0.002084] CPU: Physical Processor ID: 0
[    0.002087] CPU: Processor Core ID: 0
[    0.002094] Last level iTLB entries: 4KB 512, 2MB 0, 4MB 0
[    0.002094] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32
[    0.002094] tlb_flushall_shift is 0x5
[    0.002164] SMP alternatives: switching to UP code
[    0.025291] Freeing SMP alternatives: 24k freed
[    0.025356] cpu 0 spinlock event irq 17
[    0.025383] Performance Events: unsupported p6 CPU model 45 no PMU driver, software events only.
[    0.025551] NMI watchdog: disabled (cpu0): hardware events not enabled
[    0.025576] Brought up 1 CPUs
[    0.028642] kworker/u:0 (14) used greatest stack depth: 5936 bytes left
[    0.028675] Grant tables using version 2 layout.
[    0.028691] Grant table initialized
[    0.047616] RTC time: 165:165:165, date: 165/165/65
[    0.047661] NET: Registered protocol family 16
[    0.048184] dca service started, version 1.12.1
[    0.048545] PCI: setting up Xen PCI frontend stub
[    0.048552] PCI: pci_cache_line_size set to 64 bytes
[    0.049543] kworker/u:0 (51) used greatest stack depth: 5472 bytes left
[    0.054147] bio: create slab <bio-0> at 0
[    0.054240] ACPI: Interpreter disabled.
[    0.054288] xen/balloon: Initialising balloon driver.
[    0.055127] xen-balloon: Initialising balloon driver.
[    0.055127] vgaarb: loaded
[    0.056125] usbcore: registered new interface driver usbfs
[    0.056162] usbcore: registered new interface driver hub
[    0.056217] usbcore: registered new device driver usb
[    0.056425] PCI: System does not support PCI
[    0.056431] PCI: System does not support PCI
[    0.056617] NetLabel: Initializing
[    0.056624] NetLabel:  domain hash size = 128
[    0.056627] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.056642] NetLabel:  unlabeled traffic allowed by default
[    0.056725] Switching to clocksource xen
[    0.056795] pnp: PnP ACPI: disabled
[    0.058698] NET: Registered protocol family 2
[    0.059805] TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
[    0.061110] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    0.061243] TCP: Hash tables configured (established 524288 bind 65536)
[    0.061281] TCP: reno registered
[    0.061304] UDP hash table entries: 2048 (order: 4, 65536 bytes)
[    0.061341] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes)
[    0.061425] NET: Registered protocol family 1
[    0.061492] RPC: Registered named UNIX socket transport module.
[    0.061498] RPC: Registered udp transport module.
[    0.061504] RPC: Registered tcp transport module.
[    0.061510] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.061518] PCI: CLS 0 bytes, default 64
[    0.061643] Trying to unpack rootfs image as initramfs...
[    0.382189] Freeing initrd memory: 362080k freed
[    0.499615] platform rtc_cmos: registered platform RTC device (no PNP device found)
[    0.499831] Machine check injector initialized
[    0.500181] microcode: CPU0 sig=0x206d2, pf=0x1, revision=0x8000020c
[    0.500229] microcode: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
[    0.500544] audit: initializing netlink socket (disabled)
[    0.500566] type=2000 audit(1343845740.901:1): initialized
[    0.515227] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.515358] VFS: Disk quotas dquot_6.5.2
[    0.515386] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.515525] NFS: Registering the id_resolver key type
[    0.515544] Key type id_resolver registered
[    0.515551] Key type id_legacy registered
[    0.515599] NTFS driver 2.1.30 [Flags: R/W].
[    0.515706] msgmni has been set to 8021
[    0.515765] SELinux:  Registering netfilter hooks
[    0.516222] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    0.516232] io scheduler noop registered
[    0.516236] io scheduler deadline registered
[    0.516243] io scheduler cfq registered (default)
[    0.516337] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    0.516442] ioatdma: Intel(R) QuickData Technology Driver 4.00
[    0.532923] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.533399] Non-volatile memory driver v1.3
[    0.533406] Linux agpgart interface v0.103
[    0.533588] [drm] Initialized drm 1.1.0 20060810
[    0.535196] brd: module loaded
[    0.535992] loop: module loaded
[    0.536344] libphy: Fixed MDIO Bus: probed
[    0.536351] tun: Universal TUN/TAP device driver, 1.6
[    0.536354] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[    0.536419] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.6.0-k
[    0.536428] ixgbevf: Copyright (c) 2009 - 2012 Intel Corporation.
[    0.536721] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    0.536729] ehci_hcd: block sizes: qh 104 qtd 96 itd 192 sitd 96
[    0.536770] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    0.536776] ohci_hcd: block sizes: ed 80 td 96
[    0.536826] uhci_hcd: USB Universal Host Controller Interface driver
[    0.536929] usbcore: registered new interface driver usblp
[    0.536977] usbcore: registered new interface driver libusual
[    0.537164] i8042: PNP: No PS/2 controller found. Probing ports directly.
[    0.538013] i8042: No controller found
[    0.538103] mousedev: PS/2 mouse device common for all mice
[    0.598349] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[    0.598404] rtc_cmos: probe of rtc_cmos failed with error -38
[    0.598559] EFI Variables Facility v0.08 2004-May-17
[    0.598701] Netfilter messages via NETLINK v0.30.
[    0.598719] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[    0.598790] ctnetlink v0.93: registering with nfnetlink.
[    0.598971] ip_tables: (C) 2000-2006 Netfilter Core Team
[    0.599007] TCP: cubic registered
[    0.599014] Initializing XFRM netlink socket
[    0.599043] NET: Registered protocol family 10
[    0.599238] ip6_tables: (C) 2000-2006 Netfilter Core Team
[    0.599374] sit: IPv6 over IPv4 tunneling driver
[    0.599589] NET: Registered protocol family 17
[    0.599618] Key type dns_resolver registered
[    0.599808] PM: Hibernation image not present or could not be loaded.
[    0.599824] registered taskstats version 1
[    0.599848] XENBUS: Device with no driver: device/vkbd/0
[    0.599852] XENBUS: Device with no driver: device/vfb/0
[    0.599856] XENBUS: Device with no driver: device/vbd/51712
[    0.599860] XENBUS: Device with no driver: device/vif/0
[    0.599886]   Magic number: 1:252:3141
[    0.600271] Freeing unused kernel memory: 704k freed
[    0.600500] Write protecting the kernel read-only data: 8192k
[    0.602437] Freeing unused kernel memory: 132k freed
[    0.602589] Freeing unused kernel memory: 340k freed
init started: BusyBox v1.14.3 (2012-08-01 13:52:44 EDT)
[    0.607489] consoletype (1056) used greatest stack depth: 5288 bytes left
Mounting directories  [  OK  ]
mount: mount point /proc/bus/usb does not exist
[    0.781044] modprobe (1085) used greatest stack depth: 5048 bytes left
mount: mount point /sys/kernel/config does not exist
[    0.785748] core_filesystem (1057) used greatest stack depth: 4968 bytes left
[    0.793721] input: Xen Virtual Keyboard as /devices/virtual/input/input0
[    0.793892] input: Xen Virtual Pointer as /devices/virtual/input/input1
[    1.010121] Initialising Xen virtual ethernet driver.
[    1.124604] blkfront: xvda: flush diskcache: enabled
[    1.126118]  xvda: xvda1 xvda2 xvda3 xvda4
[    1.239316] udevd (1128): /proc/1128/oom_adj is deprecated, please use /proc/1128/oom_score_adj instead.
udevd-work[1130]: error opening ATTR{/sys/devices/system/cpu/cpu0/online} for writing: No such file or directory

[    1.395080] ip (1408) used greatest stack depth: 3912 bytes left
Waiting for devices [  OK  ]
Waiting for init.pre_custom [  OK  ]
Waiting for fb [  OK  ]
Starting..[/dev/fb0]
/dev/fb0: len:0
/dev/fb0: bits/pixel32
(7f44ddbc2000): Writting .. [800:600]
Done!
FATAL: Module agpgart_intel not found.
[    1.652131] [drm] radeon kernel modesetting enabled.
WARNING: Error inserting video (/lib/modules/3.5.0upstream-08854-g444fa66/kernel/drivers/acpi/video.ko): No such device
WARNING: Error inserting mxm_wmi (/lib/modules/3.5.0upstream-08854-g444fa66/kernel/drivers/platform/x86/mxm-wmi.ko): No such device
WARNING: Error inserting drm_kms_helper (/lib/modules/3.5.0upstream-08854-g444fa66/kernel/drivers/gpu/drm/drm_kms_helper.ko): No such device
WARNING: Error inserting ttm (/lib/modules/3.5.0upstream-08854-g444fa66/kernel/drivers/gpu/drm/ttm/ttm.ko): No such device
FATAL: Error inserting nouveau (/lib/modules/3.5.0upstream-08854-g444fa66/kernel/drivers/gpu/drm/nouveau/nouveau.ko): No such device
[    1.660288] Console: switching to colour frame buffer device 100x37
WARNING: Error inserting drm_kms_helper (/lib/modules/3.5.0upstream-08854-g444fa66/kernel/drivers/gpu/drm/drm_kms_helper.ko): No such device
FATAL: Error inserting i915 (/lib/modules/3.5.0upstream-08854-g444fa66/kernel/drivers/gpu/drm/i915/i915.ko): No such device
Starting..[/dev/fb0]
/dev/fb0: len:0
/dev/fb0: bits/pixel32
(7fb8669cc000): Writting .. [800:600]
Done!
VGA: 0000:
Waiting for network [  OK  ]
Bringing up loopback interface:  [  OK  ]
Bringing up interface eth0:  [    1.908592] BUG: unable to handle kernel NULL pointer dereference at 0000000000000010
[    1.908643] IP: [<ffffffffa0037750>] xennet_poll+0x980/0xec0 [xen_netfront]
[    1.908703] PGD ea1df067 PUD e8ada067 PMD 0 
[    1.908774] Oops: 0000 [#1] SMP 
[    1.908797] Modules linked in: fbcon tileblit font radeon bitblit softcursor ttm drm_kms_helper crc32c_intel xen_blkfront xen_netfront xen_fbfront fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront xenfs xen_privcmd
[    1.908938] CPU 0 
[    1.908950] Pid: 2165, comm: ip Not tainted 3.5.0upstream-08854-g444fa66 #1  
[    1.908983] RIP: e030:[<ffffffffa0037750>]  [<ffffffffa0037750>] xennet_poll+0x980/0xec0 [xen_netfront]
[    1.909029] RSP: e02b:ffff8800ffc03db8  EFLAGS: 00010282
[    1.909055] RAX: ffff8800ea010140 RBX: ffff8800f00e86c0 RCX: 000000000000009a
[    1.909055] RDX: 0000000000000040 RSI: 000000000000005a RDI: ffff8800fa7dee80
[    1.909055] RBP: ffff8800ffc03ee8 R08: ffff8800f00e86d8 R09: ffff8800ea010000
[    1.909055] R10: dead000000200200 R11: dead000000100100 R12: ffff8800fa7dee80
[    1.909055] R13: 000000000000005a R14: ffff8800fa7dee80 R15: 0000000000000200
[    1.909055] FS:  00007fbafc188700(0000) GS:ffff8800ffc00000(0000) knlGS:0000000000000000
[    1.909055] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[    1.909055] CR2: 0000000000000010 CR3: 00000000ea108000 CR4: 0000000000002660
[    1.909055] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    1.909055] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[    1.909055] Process ip (pid: 2165, threadinfo ffff8800ea0f2000, task ffff8800fa783040)
[    1.909055] Stack:
[    1.909055]  ffff8800e27e5040 ffff8800ffc03e88 ffff8800ffc03e68 ffff8800ffc03e48
[    1.909055]  7fffffffffffffff ffff8800ffc03e00 ffff8800e27e5040 ffff8800f00e86d8
[    1.909055]  ffff8800ffc03eb0 00000040ffffffff ffff8800f00e8000 00000000ffc03e30
[    1.909055] Call Trace:
[    1.909055]  <IRQ> 
[    1.909055]  [<ffffffff81066028>] ? pvclock_clocksource_read+0x58/0xd0
[    1.909055]  [<ffffffff81486352>] net_rx_action+0x112/0x240
[    1.909055]  [<ffffffff8107f319>] __do_softirq+0xb9/0x190
[    1.909055]  [<ffffffff815d8d7c>] call_softirq+0x1c/0x30
[    1.909055]  <EOI> 
[    1.909055]  [<ffffffff8103a435>] do_softirq+0x65/0xa0
[    1.909055]  [<ffffffff8107f834>] local_bh_enable_ip+0x94/0xa0
[    1.909055]  [<ffffffff815d11a4>] _raw_spin_unlock_bh+0x24/0x30
[    1.909055]  [<ffffffffa0036d44>] xennet_open+0x54/0xe0 [xen_netfront]
[    1.909055]  [<ffffffff81481dcf>] __dev_open+0xbf/0x120
[    1.909055]  [<ffffffff8148022c>] __dev_change_flags+0x9c/0x180
[    1.909055]  [<ffffffff81481cc3>] dev_change_flags+0x23/0x70
[    1.909055]  [<ffffffff81491062>] do_setlink+0x1c2/0xa10
[    1.909055]  [<ffffffff812b5d74>] ? nla_parse+0x34/0x110
[    1.909055]  [<ffffffff81494005>] rtnl_newlink+0x3a5/0x5c0
[    1.909055]  [<ffffffff812541c4>] ? selinux_capable+0x34/0x50
[    1.909055]  [<ffffffff81250223>] ? security_capable+0x13/0x20
[    1.909055]  [<ffffffff81491e07>] rtnetlink_rcv_msg+0x2c7/0x330
[    1.909055]  [<ffffffff810a18b7>] ? __might_sleep+0xe7/0x100
[    1.909055]  [<ffffffff81149a52>] ? kmem_cache_alloc_node+0x82/0x1d0
[    1.909055]  [<ffffffff8147a00c>] ? __skb_recv_datagram+0x11c/0x2f0
[    1.909055]  [<ffffffff81491b40>] ? rtnetlink_rcv+0x30/0x30
[    1.909055]  [<ffffffff814a9c89>] netlink_rcv_skb+0x99/0xc0
[    1.909055]  [<ffffffff81491b30>] rtnetlink_rcv+0x20/0x30
[    1.909055]  [<ffffffff814a9998>] netlink_unicast+0x1a8/0x220
[    1.909055]  [<ffffffff814aa535>] netlink_sendmsg+0x205/0x300
[    1.909055]  [<ffffffff8146ce19>] sock_sendmsg+0xb9/0xf0
[    1.909055]  [<ffffffff8146c51e>] ? copy_from_user+0x3e/0x50
[    1.909055]  [<ffffffff8146c576>] ? move_addr_to_kernel+0x46/0x80
[    1.909055]  [<ffffffff810a18b7>] ? __might_sleep+0xe7/0x100
[    1.909055]  [<ffffffff8146dd2d>] __sys_sendmsg+0x3dd/0x400
[    1.909055]  [<ffffffff8112c751>] ? handle_mm_fault+0x261/0x380
[    1.909055]  [<ffffffff815d4cd0>] ? do_page_fault+0x250/0x4f0
[    1.909055]  [<ffffffff8114a587>] ? kmem_cache_alloc+0x1a7/0x1f0
[    1.909055]  [<ffffffff811311a4>] ? do_brk+0x1b4/0x350
[    1.909055]  [<ffffffff8146df04>] sys_sendmsg+0x44/0x80
[    1.909055]  [<ffffffff815d7bf9>] system_call_fastpath+0x16/0x1b
[    1.909055] Code: 44 00 00 41 80 8c 24 a8 00 00 00 04 e9 2f ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 41 8b 84 24 c8 00 00 00 49 03 84 24 d0 00 00 00 <80> 3c 25 10 00 00 00 00 48 8d 50 30 74 0f 48 83 3c 25 08 00 00 
[    1.909055] RIP  [<ffffffffa0037750>] xennet_poll+0x980/0xec0 [xen_netfront]
[    1.909055]  RSP <ffff8800ffc03db8>
[    1.909055] CR2: 0000000000000010
[    1.947298] ---[ end trace 3f4ba742dffbe90d ]---
[    1.947824] Kernel panic - not syncing: Fatal exception in interrupt