From mboxrd@z Thu Jan 1 00:00:00 1970 From: Marek =?utf-8?Q?Marczykowski-G=C3=B3recki?= Subject: Re: xen-netfront crash when detaching network while some network activity Date: Fri, 22 Jan 2016 20:23:43 +0100 Message-ID: <20160122192343.GL31058@mail-itl> References: <20150522114932.GC8664@mail-itl> <55645140.1050209@citrix.com> <20150526220312.GA1358@mail-itl> <20151021185734.GD31646@mail-itl> <20151117024515.GN976@mail-itl> <20151201220042.GC22107@char.us.oracle.com> <20151201223258.GH1337@mail-itl> <20160120215914.GA6384@char.us.oracle.com> <56A0CF78.1090901@oracle.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="RD6GsZsdEJvsf78O" Cc: Konrad Rzeszutek Wilk , David Vrabel , Boris Ostrovsky , Annie Li , xen-devel , netdev@vger.kernel.org To: Joao Martins Return-path: Received: from out3-smtp.messagingengine.com ([66.111.4.27]:37335 "EHLO out3-smtp.messagingengine.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754243AbcAVTXu (ORCPT ); Fri, 22 Jan 2016 14:23:50 -0500 Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 667E821554 for ; Fri, 22 Jan 2016 14:23:49 -0500 (EST) Content-Disposition: inline In-Reply-To: <56A0CF78.1090901@oracle.com> Sender: netdev-owner@vger.kernel.org List-ID: --RD6GsZsdEJvsf78O Content-Type: multipart/mixed; boundary="FCF9ydJxlAJsfRTu" Content-Disposition: inline --FCF9ydJxlAJsfRTu Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jan 21, 2016 at 12:30:48PM +0000, Joao Martins wrote: >=20 >=20 > On 01/20/2016 09:59 PM, Konrad Rzeszutek Wilk wrote: > > On Tue, Dec 01, 2015 at 11:32:58PM +0100, Marek Marczykowski-G=C3=B3rec= ki wrote: > >> On Tue, Dec 01, 2015 at 05:00:42PM -0500, Konrad Rzeszutek Wilk wrote: > >>> On Tue, Nov 17, 2015 at 03:45:15AM +0100, Marek Marczykowski-G=C3=B3r= ecki wrote: > >>>> On Wed, Oct 21, 2015 at 08:57:34PM +0200, Marek Marczykowski-G=C3=B3= recki wrote: > >>>>> On Wed, May 27, 2015 at 12:03:12AM +0200, Marek Marczykowski-G=C3= =B3recki wrote: > >>>>>> On Tue, May 26, 2015 at 11:56:00AM +0100, David Vrabel wrote: > >>>>>>> On 22/05/15 12:49, Marek Marczykowski-G=C3=B3recki wrote: > >>>>>>>> Hi all, > >>>>>>>> > >>>>>>>> I'm experiencing xen-netfront crash when doing xl network-detach= while > >>>>>>>> some network activity is going on at the same time. It happens o= nly when > >>>>>>>> domU has more than one vcpu. Not sure if this matters, but the b= ackend > >>>>>>>> is in another domU (not dom0). I'm using Xen 4.2.2. It happens o= n kernel > >>>>>>>> 3.9.4 and 4.1-rc1 as well. > >>>>>>>> > >>>>>>>> Steps to reproduce: > >>>>>>>> 1. Start the domU with some network interface > >>>>>>>> 2. Call there 'ping -f some-IP' > >>>>>>>> 3. Call 'xl network-detach NAME 0' > >>> > >>> Do you see this all the time or just on occassions? > >> > >> Using above procedure - all the time. > >> > >>> I tried to reproduce it and couldn't see it. Is your VM an PV or HVM? > >> > >> PV, started by libvirt. This may have something to do, the problem did= n't > >> existed on older Xen (4.1) and started by xl. I'm not sure about kernel > >> version there, but I think I've tried there 3.18 too, which has this > >> problem. > >> > >> But I don't see anything special in domU config file (neither backend > >> nor frontend) - it may be some libvirt default. If that's really the > >> cause. Can I (and how) get any useful information about that? > >=20 > > libvirt naturally does some libxl calls, and they may be different. > >=20 > > Any chance you could give me an idea of: > > - What commands you use in libvirt? > > - Do you use a bond or bridge? > > - What version of libvirt you are using? > >=20 > > Thanks! > > CC-ing Joao just in case he has seen this. > >> > Hm, So far I couldn't reproduce the issue with upstream Xen/linux/libvirt= , using > both libvirt or plain xl (both on a bridge setup) and also irrespective o= f the > both load and direction of traffic (be it a ping flood, pktgen with min. > sized packets or iperf). I've ran the test again, on vanilla 4.4 and collected some info: - xenstore dump of frontend (xs-frontend-before.txt) - xenstore dump of backend (xs-backend-before.txt) - kernel messages (console output) (console.log) - kernel config (config-4.4) - libvirt config of that domain (netdebug.conf) Versions: - kernel 4.4 (frontend), 4.2.8 (backend) - libvirt 1.2.20 - xen 4.6.0 In backend domain there is no bridge or anything like that - only routing. The same in frontend - nothing fancy - just IP set on eth0 there. Steps to reproduce were the same: - start frontend domain (virsh create ...) - call ping -f - xl network-detach NAME 0 Note that the crash doesn't happen with attached patch applied (as noted in mail on Oct 21), but I have no idea whether is it a proper fix, or just prevents the crash by a coincidence. --=20 Best Regards, Marek Marczykowski-G=C3=B3recki Invisible Things Lab A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? --FCF9ydJxlAJsfRTu Content-Type: text/plain; charset=utf-8 Content-Disposition: attachment; filename="console.log" Content-Transfer-Encoding: quoted-printable [ 0.000000] x86/PAT: Configuration [0-7]: WB WT UC- UC WC WP UC UC= =0D [ 0.000000] Initializing cgroup subsys cpuset=0D [ 0.000000] Initializing cgroup subsys cpu=0D [ 0.000000] Initializing cgroup subsys cpuacct=0D [ 0.000000] Linux version 4.4.0-1.pvops.qubes.x86_64 (user@devel-3rdpart= y) (gcc version 4.9.2 20150212 (Red Hat 4.9.2-6) (GCC) ) #20 SMP Fri Jan 22= 00:39:29 CET 2016=0D [ 0.000000] Command line: root=3D/dev/mapper/dmroot ro nomodeset console= =3Dhvc0 rd_NO_PLYMOUTH 3 rd.break=0D [ 0.000000] x86/fpu: Legacy x87 FPU detected.=0D [ 0.000000] x86/fpu: Using 'lazy' FPU context switches.=0D [ 0.000000] ACPI in unprivileged domain disabled=0D [ 0.000000] Released 0 page(s)=0D [ 0.000000] e820: BIOS-provided physical RAM map:=0D [ 0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable=0D [ 0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved=0D [ 0.000000] Xen: [mem 0x0000000000100000-0x00000000f9ffffff] usable=0D [ 0.000000] NX (Execute Disable) protection: active=0D [ 0.000000] DMI not present or invalid.=0D [ 0.000000] Hypervisor detected: Xen=0D [ 0.000000] e820: last_pfn =3D 0xfa000 max_arch_pfn =3D 0x400000000=0D [ 0.000000] MTRR: Disabled=0D [ 0.000000] RAMDISK: [mem 0x02030000-0x027c6fff]=0D [ 0.000000] NUMA turned off=0D [ 0.000000] Faking a node at [mem 0x0000000000000000-0x00000000f9ffffff]= =0D [ 0.000000] NODE_DATA(0) allocated [mem 0x18837000-0x18848fff]=0D [ 0.000000] Zone ranges:=0D [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff]=0D [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000f9ffffff]=0D [ 0.000000] Normal empty=0D [ 0.000000] Movable zone start for each node=0D [ 0.000000] Early memory node ranges=0D [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009ffff]=0D [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000f9ffffff]=0D [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x00000000f9fff= fff]=0D [ 0.000000] p2m virtual area at ffffc90000000000, size is 40000000=0D [ 0.000000] Remapped 0 page(s)=0D [ 0.000000] SFI: Simple Firmware Interface v0.81 http://simplefirmware.o= rg=0D [ 0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs=0D [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff]=0D [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000fffff]=0D [ 0.000000] e820: [mem 0xfa000000-0xffffffff] available for PCI devices= =0D [ 0.000000] Booting paravirtualized kernel on Xen=0D [ 0.000000] Xen version: 4.6.0 (preserve-AD)=0D [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0= xffffffff, max_idle_ns: 1910969940391419 ns=0D [ 0.000000] setup_percpu: NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:2 nr_= node_ids:1=0D [ 0.000000] PERCPU: Embedded 33 pages/cpu @ffff880013c00000 s98264 r8192= d28712 u1048576=0D [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Tota= l pages: 1007882=0D [ 0.000000] Policy zone: DMA32=0D [ 0.000000] Kernel command line: root=3D/dev/mapper/dmroot ro nomodeset = console=3Dhvc0 rd_NO_PLYMOUTH 3 rd.break=0D [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)=0D [ 0.000000] Memory: 302592K/4095612K available (6670K kernel code, 1222K= rwdata, 2980K rodata, 1452K init, 1424K bss, 3793020K reserved, 0K cma-res= erved)=0D [ 0.000000] SLUB: HWalign=3D64, Order=3D0-3, MinObjects=3D0, CPUs=3D2, N= odes=3D1=0D [ 0.000000] Hierarchical RCU implementation.=0D [ 0.000000] Build-time adjustment of leaf fanout to 64.=0D [ 0.000000] RCU restricting CPUs from NR_CPUS=3D64 to nr_cpu_ids=3D2.=0D [ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=3D64, nr_cpu_ids= =3D2=0D [ 0.000000] Using NULL legacy PIC=0D [ 0.000000] NR_IRQS:4352 nr_irqs:48 0=0D [ 0.000000] xen:events: Using FIFO-based ABI=0D [ 0.000000] Offload RCU callbacks from all CPUs=0D [ 0.000000] Offload RCU callbacks from CPUs: 0-1.=0D [ 0.000000] Console: colour dummy device 80x25=0D [ 0.000000] console [tty0] enabled=0D [ 0.000000] console [hvc0] enabled=0D [ 0.000000] clocksource: xen: mask: 0xffffffffffffffff max_cycles: 0x1cd= 42e4dffb, max_idle_ns: 881590591483 ns=0D [ 0.000000] installing Xen timer for CPU 0=0D [ 0.000000] tsc: Detected 1995.104 MHz processor=0D [ 0.001000] Calibrating delay loop (skipped), value calculated using tim= er frequency.. 3990.20 BogoMIPS (lpj=3D1995104)=0D [ 0.001000] pid_max: default: 32768 minimum: 301=0D [ 0.001000] Security Framework initialized=0D [ 0.001611] Dentry cache hash table entries: 524288 (order: 10, 4194304 = bytes)=0D [ 0.004023] Inode-cache hash table entries: 262144 (order: 9, 2097152 by= tes)=0D [ 0.005197] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes)= =0D [ 0.005237] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 b= ytes)=0D [ 0.005788] Initializing cgroup subsys io=0D [ 0.005803] Initializing cgroup subsys memory=0D [ 0.005821] Initializing cgroup subsys devices=0D [ 0.005832] Initializing cgroup subsys freezer=0D [ 0.005843] Initializing cgroup subsys net_cls=0D [ 0.005853] Initializing cgroup subsys perf_event=0D [ 0.005863] Initializing cgroup subsys net_prio=0D [ 0.005873] Initializing cgroup subsys hugetlb=0D [ 0.005956] CPU: Physical Processor ID: 0=0D [ 0.005963] CPU: Processor Core ID: 1=0D [ 0.005977] Last level iTLB entries: 4KB 128, 2MB 4, 4MB 4=0D [ 0.005983] Last level dTLB entries: 4KB 256, 2MB 0, 4MB 32, 1GB 0=0D [ 0.039457] ftrace: allocating 25787 entries in 101 pages=0D [ 0.047128] Could not initialize VPMU for cpu 0, error -95=0D [ 0.047220] Performance Events: unsupported p6 CPU model 15 no PMU drive= r, software events only.=0D [ 0.047979] NMI watchdog: disabled (cpu0): hardware events not enabled=0D [ 0.047990] NMI watchdog: Shutting down hard lockup detector on all cpus= =0D [ 0.048139] SMP alternatives: switching to SMP code=0D [ 0.078846] installing Xen timer for CPU 1=0D [ 0.079180] x86: Booted up 1 node, 2 CPUs=0D [ 0.079316] devtmpfs: initialized=0D [ 0.083816] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xfffffff= f, max_idle_ns: 1911260446275000 ns=0D [ 0.084096] atomic64_test: passed for x86-64 platform with CX8 and with = SSE=0D [ 0.084108] pinctrl core: initialized pinctrl subsystem=0D [ 0.104107] RTC time: 165:165:165, date: 165/165/65=0D [ 0.110246] NET: Registered protocol family 16=0D [ 0.110284] xen:grant_table: Grant tables using version 1 layout=0D [ 0.110310] Grant table initialized=0D [ 0.111010] PCI: setting up Xen PCI frontend stub=0D [ 0.115090] ACPI: Interpreter disabled.=0D [ 0.115090] xen:balloon: Initialising balloon driver=0D [ 0.181042] xen_balloon: Initialising balloon driver=0D [ 0.181110] vgaarb: loaded=0D [ 0.182053] SCSI subsystem initialized=0D [ 0.182165] dmi: Firmware registration failed.=0D [ 0.183040] PCI: System does not support PCI=0D [ 0.183050] PCI: System does not support PCI=0D [ 0.183237] NetLabel: Initializing=0D [ 0.183248] NetLabel: domain hash size =3D 128=0D [ 0.183254] NetLabel: protocols =3D UNLABELED CIPSOv4=0D [ 0.183279] NetLabel: unlabeled traffic allowed by default=0D [ 0.185019] clocksource: Switched to clocksource xen=0D [ 0.195845] pnp: PnP ACPI: disabled=0D [ 0.200085] NET: Registered protocol family 2=0D [ 0.200441] TCP established hash table entries: 32768 (order: 6, 262144 = bytes)=0D [ 0.200691] TCP bind hash table entries: 32768 (order: 7, 524288 bytes)= =0D [ 0.200901] TCP: Hash tables configured (established 32768 bind 32768)=0D [ 0.200975] UDP hash table entries: 2048 (order: 4, 65536 bytes)=0D [ 0.201057] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes)=0D [ 0.201180] NET: Registered protocol family 1=0D [ 0.201284] Unpacking initramfs...=0D [ 0.219827] Freeing initrd memory: 7772K (ffff880002030000 - ffff8800027= c7000)=0D [ 0.220918] futex hash table entries: 512 (order: 3, 32768 bytes)=0D [ 0.220977] audit: initializing netlink subsys (disabled)=0D [ 0.221036] audit: type=3D2000 audit(1453421870.210:1): initialized=0D [ 0.221566] HugeTLB registered 2 MB page size, pre-allocated 0 pages=0D [ 0.224154] zbud: loaded=0D [ 0.224502] VFS: Disk quotas dquot_6.6.0=0D [ 0.224567] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 byte= s)=0D [ 0.225281] Key type big_key registered=0D [ 0.230104] Block layer SCSI generic (bsg) driver version 0.4 loaded (ma= jor 253)=0D [ 0.230168] io scheduler noop registered=0D [ 0.230177] io scheduler deadline registered=0D [ 0.230239] io scheduler cfq registered (default)=0D [ 0.230356] pci_hotplug: PCI Hot Plug PCI Core version: 0.5=0D [ 0.230371] pciehp: PCI Express Hot Plug Controller Driver version: 0.4= =0D [ 0.230721] xen:xen_evtchn: Event-channel device installed=0D [ 0.231501] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled=0D [ 0.232242] Non-volatile memory driver v1.3=0D [ 0.232585] loop: module loaded=0D [ 0.283555] blkfront: xvda: flush diskcache: enabled; persistent grants:= enabled; indirect descriptors: enabled;=0D [ 0.287743] tun: Universal TUN/TAP device driver, 1.6=0D [ 0.287754] tun: (C) 1999-2004 Max Krasnyansky =0D [ 0.287959] xen_netfront: Initialising Xen virtual ethernet driver=0D [ 0.291795] blkfront: xvdb: flush diskcache: enabled; persistent grants:= enabled; indirect descriptors: enabled;=0D [ 0.295457] i8042: PNP: No PS/2 controller found. Probing ports directly= =2E=0D [ 1.314402] i8042: No controller found=0D [ 1.314436] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x398= 43a855e2, max_idle_ns: 881590688655 ns=0D [ 1.314792] mousedev: PS/2 mouse device common for all mice=0D [ 1.315247] device-mapper: uevent: version 1.0.3=0D [ 1.315492] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised:= dm-devel@redhat.com=0D [ 1.315696] dmi-sysfs: dmi entry is absent.=0D [ 1.315738] hidraw: raw HID events driver (C) Jiri Kosina=0D [ 1.315852] drop_monitor: Initializing network drop monitor service=0D [ 1.315956] Initializing XFRM netlink socket=0D [ 1.316215] NET: Registered protocol family 10=0D [ 1.316696] mip6: Mobile IPv6=0D [ 1.316710] NET: Registered protocol family 17=0D [ 1.317235] registered taskstats version 1=0D [ 1.317290] zswap: loaded using pool lzo/zbud=0D [ 1.386055] blkfront: xvdc: flush diskcache: enabled; persistent grants:= enabled; indirect descriptors: enabled;=0D [ 1.392405] blkfront: xvdd: flush diskcache: enabled; persistent grants:= enabled; indirect descriptors: enabled;=0D [ 1.417232] Magic number: 1:252:3141=0D [ 1.417399] console [netcon0] enabled=0D [ 1.417417] netconsole: network logging started=0D [ 1.417441] hctosys: unable to open rtc device (rtc0)=0D [ 1.420219] Freeing unused kernel memory: 1452K (ffffffff81d33000 - ffff= ffff81e9e000)=0D [ 1.420258] Write protecting the kernel read-only data: 12288k=0D [ 1.434129] Freeing unused kernel memory: 1508K (ffff880001687000 - ffff= 880001800000)=0D [ 1.435108] Freeing unused kernel memory: 1116K (ffff880001ae9000 - ffff= 880001c00000)=0D Qubes initramfs script here:=0D [ 1.440455] random: modprobe urandom read with 33 bits of entropy availa= ble=0D modprobe: chdir(4.4.0-1.pvops.qubes.x86_64): No such file or directory=0D modprobe: chdir(4.4.0-1.pvops.qubes.x86_64): No such file or directory=0D Qubes: Cannot load Xen Block Frontend...=0D Waiting for /dev/xvda* devices...=0D Qubes: Doing COW setup for AppVM...=0D sfdisk: Checking that no-one is using this disk right now ...=0D sfdisk: OK=0D sfdisk: /dev/xvdc: unrecognized partition table type=0D sfdisk: No partitions found=0D sfdisk: Warning: partition 1 does not end at a cylinder boundary=0D sfdisk: Warning: partition 2 does not start at a cylinder boundary=0D sfdisk: Warning: partition 2 does not end at a cylinder boundary=0D sfdisk: Warning: no primary partition is marked bootable (active)=0D This does not matter for LILO, but the DOS MBR will not boot this disk.=0D [ 1.448877] xvdc: xvdc1 xvdc2=0D sfdisk: If you created or changed a DOS partition, /dev/foo7, say, then use= dd(1)=0D to zero the first 512 bytes: dd if=3D/dev/zero of=3D/dev/foo7 bs=3D512 cou= nt=3D1=0D (See fdisk(8).)=0D Setting up swapspace version 1, size =3D 1073737728 bytes=0D UUID=3Dda20ab8f-e3b2-460f-8ea1-c77204a0f63d=0D Qubes: done.=0D modprobe: chdir(4.4.0-1.pvops.qubes.x86_64): No such file or directory=0D [ 1.464947] EXT4-fs (dm-0): couldn't mount as ext3 due to feature incomp= atibilities=0D [ 1.465108] EXT4-fs (dm-0): couldn't mount as ext2 due to feature incomp= atibilities=0D [ 1.466566] EXT4-fs (dm-0): mounted filesystem with ordered data mode. O= pts: (null)=0D Waiting for /dev/xvdd device...=0D [ 1.471024] EXT4-fs (xvdd): mounting ext3 file system using the ext4 sub= system=0D [ 1.471931] EXT4-fs (xvdd): mounted filesystem with ordered data mode. O= pts: (null)=0D [ 1.474541] EXT4-fs (dm-0): re-mounted. Opts: data=3Dordered=0D [ 1.484119] EXT4-fs (dm-0): re-mounted. Opts: data=3Dordered=0D mount: mounting /tmp/modules/4.4.0-1.pvops.qubes.x86_64 on /sysroot/lib/mod= ules/4.4.0-1.pvops.qubes.x86_64 failed: No such file or directory=0D [ 1.578686] systemd[1]: systemd 216 running in system mode. (+PAM +AUDIT= +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNU= TLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN)=0D [ 1.578778] systemd[1]: Detected virtualization 'xen'.=0D [ 1.578793] systemd[1]: Detected architecture 'x86-64'.=0D =0D Welcome to .[0;34mFedora 21 (Twenty One).[0m!=0D =0D [ 1.579269] systemd[1]: No hostname configured.=0D [ 1.579287] systemd[1]: Set hostname to .=0D [ 1.643346] systemd[1]: Configuration file /usr/lib/systemd/system/audit= d.service is marked world-inaccessible. This has no effect as configuration= data is accessible via APIs without restrictions. Proceeding anyway.=0D [ 1.673346] systemd[1]: Starting Forward Password Requests to Wall Direc= tory Watch.=0D [ 1.673475] systemd[1]: Started Forward Password Requests to Wall Direct= ory Watch.=0D [ 1.673505] systemd[1]: Expecting device dev-hvc0.device...=0D Expecting device dev-hvc0.device...=0D [ 1.673685] systemd[1]: Starting Remote File Systems.=0D [.[32m OK .[0m] Reached target Remote File Systems.=0D [ 1.673773] systemd[1]: Reached target Remote File Systems.=0D [ 1.673823] systemd[1]: Starting Arbitrary Executable File Formats File = System Automount Point.=0D [.[32m OK .[0m] Set up automount Arbitrary Executab...ats File System Aut= omount Point.=0D [ 1.674059] systemd[1]: Set up automount Arbitrary Executable File Forma= ts File System Automount Point.=0D [ 1.674091] systemd[1]: Starting Encrypted Volumes.=0D [.[32m OK .[0m] Reached target Encrypted Volumes.=0D [ 1.674175] systemd[1]: Reached target Encrypted Volumes.=0D [ 1.674201] systemd[1]: Expecting device dev-xvdc1.device...=0D Expecting device dev-xvdc1.device...=0D [ 1.674287] systemd[1]: Starting Root Slice.=0D [.[32m OK .[0m] Created slice Root Slice.=0D [ 1.690630] systemd[1]: Created slice Root Slice.=0D [ 1.690661] systemd[1]: Starting Delayed Shutdown Socket.=0D [.[32m OK .[0m] Listening on Delayed Shutdown Socket.=0D [ 1.690788] systemd[1]: Listening on Delayed Shutdown Socket.=0D [ 1.690813] systemd[1]: Starting /dev/initctl Compatibility Named Pipe.= =0D [.[32m OK .[0m] Listening on /dev/initctl Compatibility Named Pipe.=0D [ 1.690938] systemd[1]: Listening on /dev/initctl Compatibility Named Pi= pe.=0D [ 1.690970] systemd[1]: Starting udev Control Socket.=0D [.[32m OK .[0m] Listening on udev Control Socket.=0D [ 1.691101] systemd[1]: Listening on udev Control Socket.=0D [ 1.691130] systemd[1]: Starting udev Kernel Socket.=0D [.[32m OK .[0m] Listening on udev Kernel Socket.=0D [ 1.691234] systemd[1]: Listening on udev Kernel Socket.=0D [ 1.691257] systemd[1]: Starting User and Session Slice.=0D [.[32m OK .[0m] Created slice User and Session Slice.=0D [ 1.691711] systemd[1]: Created slice User and Session Slice.=0D [ 1.691745] systemd[1]: Starting Journal Socket.=0D [.[32m OK .[0m] Listening on Journal Socket.=0D [ 1.691894] systemd[1]: Listening on Journal Socket.=0D [ 1.691942] systemd[1]: Starting System Slice.=0D [.[32m OK .[0m] Created slice System Slice.=0D [ 1.692393] systemd[1]: Created slice System Slice.=0D [ 1.692442] systemd[1]: Mounting Temporary Directory...=0D Mounting Temporary Directory...=0D [ 1.692722] systemd[1]: tmp.mount: Directory /tmp to mount over is not e= mpty, mounting anyway.=0D [ 1.694761] systemd[1]: Starting Journal Socket (/dev/log).=0D [ 1.696712] systemd[1]: Starting udev Coldplug all Devices...=0D Starting udev Coldplug all Devices...=0D [ 1.699255] systemd[1]: Mounting Huge Pages File System...=0D Mounting Huge Pages File System...=0D [ 1.701608] systemd[1]: Mounting Debug File System...=0D Mounting Debug File System...=0D [ 1.704788] systemd[1]: Mounting POSIX Message Queue File System...=0D Mounting POSIX Message Queue File System...=0D [ 1.719399] systemd[1]: Started Create list of required static device no= des for the current kernel.=0D [ 1.719508] systemd[1]: Starting system-serial\x2dgetty.slice.=0D [.[32m OK .[0m] Created slice system-serial\x2dgetty.slice.=0D [ 1.720294] systemd[1]: Created slice system-serial\x2dgetty.slice.=0D [ 1.720357] systemd[1]: Started Collect Read-Ahead Data.=0D [ 1.720401] systemd[1]: Started Replay Read-Ahead Data.=0D [ 1.720437] systemd[1]: Starting File System Check on Root Device...=0D Starting File System Check on Root Device...=0D [ 1.727937] systemd[1]: Starting Load Kernel Modules...=0D Starting Load Kernel Modules...=0D [ 1.730073] systemd[1]: Starting Setup Virtual Console...=0D Starting Setup Virtual Console...=0D [ 1.745613] systemd[1]: Starting Load legacy module configuration...=0D Starting Load legacy module configuration...=0D [ 1.764453] systemd[1]: Started Set Up Additional Binary Formats.=0D [ 1.764540] systemd[1]: Starting Slices.=0D [.[32m OK .[0m] Reached target Slices.=0D [ 1.764610] systemd[1]: Reached target Slices.=0D [.[32m OK .[0m] Mounted POSIX Message Queue File System.=0D [ 1.765640] systemd[1]: Mounted POSIX Message Queue File System.=0D [.[32m OK .[0m] Mounted Debug File System.=0D [ 1.765721] systemd[1]: Mounted Debug File System.=0D [.[32m OK .[0m] Mounted Huge Pages File System.=0D [ 1.765779] systemd[1]: Mounted Huge Pages File System.=0D [.[32m OK .[0m] Mounted Temporary Directory.=0D [ 1.765860] systemd[1]: Mounted Temporary Directory.=0D [.[32m OK .[0m] Listening on Journal Socket (/dev/log).=0D [ 1.766401] systemd[1]: Listening on Journal Socket (/dev/log).=0D [ 1.768972] systemd[1]: systemd-modules-load.service: main process exite= d, code=3Dexited, status=3D1/FAILURE=0D [.[1;31mFAILED.[0m] Failed to start Load Kernel Modules.=0D See "systemctl status systemd-modules-load.service" for details.=0D [ 1.769555] systemd[1]: Failed to start Load Kernel Modules.=0D [ 1.769579] systemd[1]: Unit systemd-modules-load.service entered failed= state.=0D [ 1.774333] systemd[1]: systemd-modules-load.service failed.=0D [.[32m OK .[0m] Started Setup Virtual Console.=0D [ 1.774886] systemd[1]: Started Setup Virtual Console.=0D [ 1.806920] systemd[1]: Mounting Configuration File System...=0D Mounting Configuration File System...=0D [ 1.809565] systemd[1]: Starting Apply Kernel Variables...=0D Starting Apply Kernel Variables...=0D [ 1.879402] systemd[1]: Mounted FUSE Control File System.=0D [ 1.879728] systemd[1]: Starting Journal Service...=0D Starting Journal Service...=0D [.[32m OK .[0m] Started udev Coldplug all Devices.=0D [ 1.898192] systemd[1]: Started udev Coldplug all Devices.=0D [ 1.923752] systemd[1]: Starting Show Plymouth Boot Screen...=0D Starting Show Plymouth Boot Screen...=0D [.[32m OK .[0m] Mounted Configuration File System.=0D [ 1.950124] systemd[1]: Mounted Configuration File System.=0D [.[32m OK .[0m] Started Load legacy module configuration.=0D [ 1.950948] systemd[1]: Started Load legacy module configuration.=0D [.[32m OK .[0m] Started Apply Kernel Variables.=0D [ 1.951688] systemd[1]: Started Apply Kernel Variables.=0D [ 1.982704] systemd-fsck[142]: /dev/mapper/dmroot: clean, 139351/655360 = files, 1285520/2621440 blocks=0D [.[32m OK .[0m] Started Journal Service.=0D [ 1.984482] systemd[1]: Started Journal Service.=0D [.[32m OK .[0m] Started File System Check on Root Device.=0D [ 2.049986] EXT4-fs (dm-0): re-mounted. Opts: (null)=0D [ 2.118620] systemd-journald[173]: Received request to flush runtime jou= rnal from PID 1=0D [ 5.367681] Adding 1048572k swap on /dev/xvdc1. Priority:-1 extents:1 a= cross:1048572k SSFS=0D Starting Remount Root and Kernel File Systems...=0D=0D [.[32m OK .[0m] Started Remount Root and Kernel File Systems.=0D=0D Starting Configure read-only root support...=0D=0D Starting Flush Journal to Persistent Storage...=0D=0D Starting Create Static Device Nodes in /dev...=0D=0D Starting Load/Save Random Seed...=0D=0D [.[32m OK .[0m] Started Configure read-only root support.=0D=0D [.[32m OK .[0m] Started Load/Save Random Seed.=0D=0D [.[32m OK .[0m] Started Create Static Device Nodes in /dev.=0D=0D Starting udev Kernel Device Manager...=0D=0D [.[32m OK .[0m] Reached target Local File Systems (Pre).=0D=0D Mounting /proc/xen...=0D=0D [.[32m OK .[0m] Mounted /proc/xen.=0D=0D [.[32m OK .[0m] Reached target Local File Systems.=0D=0D Starting Qubes DB agent...=0D=0D Starting Tell Plymouth To Write Out Runtime Data...=0D=0D [.[32m OK .[0m] Started Tell Plymouth To Write Out Runtime Data.=0D=0D [.[32m OK .[0m] Started Qubes DB agent.=0D=0D Starting Qubes Random Seed...=0D=0D Starting Init Qubes Services settings...=0D=0D [.[32m OK .[0m] Started udev Kernel Device Manager.=0D=0D [.[32m OK .[0m] Started Qubes Random Seed.=0D=0D [.[32m OK .[0m] Started Flush Journal to Persistent Storage.=0D=0D Starting Create Volatile Files and Directories...=0D=0D [.[32m OK .[0m] Found device /dev/hvc0.=0D=0D [.[32m OK .[0m] Started Create Volatile Files and Directories.=0D=0D Starting Update UTMP about System Boot/Shutdown...=0D=0D [.[32m OK .[0m] Started Update UTMP about System Boot/Shutdown.=0D=0D =2E%G[.[32m OK .[0m] Started Init Qubes Services settings.=0D=0D [.[32m OK .[0m] Found device /dev/xvdc1.=0D=0D Activating swap /dev/xvdc1...=0D=0D [.[32m OK .[0m] Activated swap /dev/xvdc1.=0D=0D [.[32m OK .[0m] Reached target Swap.=0D=0D [.[32m OK .[0m] Reached target System Initialization.=0D=0D [.[32m OK .[0m] Listening on CUPS Printing Service Sockets.=0D=0D Starting Manage Sound Card State (restore and store)...=0D=0D [.[32m OK .[0m] Started Manage Sound Card State (restore and store).=0D=0D [.[32m OK .[0m] Reached target Timers.=0D=0D [.[32m OK .[0m] Listening on D-Bus System Message Bus Socket.=0D=0D [.[32m OK .[0m] Reached target Sockets.=0D=0D [.[32m OK .[0m] Started Show Plymouth Boot Screen.=0D=0D [.[32m OK .[0m] Reached target Paths.=0D=0D [.[32m OK .[0m] Reached target Basic System.=0D=0D Starting ABRT Automated Bug Reporting Tool...=0D=0D [.[32m OK .[0m] Started ABRT Automated Bug Reporting Tool.=0D=0D Starting Initialize and mount /rw and /home...=0D=0D Starting Initialize and mount /rw and /home...=0D=0D Starting ABRT kernel log watcher...=0D=0D [.[32m OK .[0m] Started ABRT kernel log watcher.=0D=0D Starting Install ABRT coredump hook...=0D=0D Starting Entropy Daemon based on the HAVEGE algorithm...=0D=0D [.[32m OK .[0m] Started Entropy Daemon based on the HAVEGE algorithm.=0D= =0D Starting Machine Check Exception Logging Daemon...=0D=0D [ 5.654686] random: nonblocking pool is initialized=0D Starting Qubes memory information reporter...=0D=0D Starting Qubes remote exec agent...=0D=0D Starting ABRT Xorg log watcher...=0D=0D [.[32m OK .[0m] Started ABRT Xorg log watcher.=0D=0D Starting Qubes base firewall settings...=0D=0D Starting Login Service...=0D=0D Starting Permit User Sessions...=0D=0D Starting LSB: Start/stop xen driver domain daemon...=0D=0D Starting D-Bus System Message Bus...=0D=0D [ 5.823870] EXT4-fs (xvdb): recovery complete=0D [.[32m OK .[0m] Started D-Bus System Message Bus.=0D=0D [ 5.831639] EXT4-fs (xvdb): mounted filesystem with ordered data mode. O= pts: discard=0D [.[32m OK .[0m] Started Initialize and mount /rw and /home.=0D=0D [.[32m OK .[0m] Started Initialize and mount /rw and /home.=0D=0D [.[32m OK .[0m] Started Install ABRT coredump hook.=0D=0D [.[32m OK .[0m] Started Machine Check Exception Logging Daemon.=0D=0D [.[32m OK .[0m] Started Qubes memory information reporter.=0D=0D [.[32m OK .[0m] Started Qubes remote exec agent.=0D=0D [.[1;31mFAILED.[0m] Failed to start Qubes base firewall settings.=0D=0D See "systemctl status qubes-iptables.service" for details.=0D=0D [.[32m OK .[0m] Started Permit User Sessions.=0D=0D [.[32m OK .[0m] Started LSB: Start/stop xen driver domain daemon.=0D=0D [.[32m OK .[0m] Started Login Service.=0D=0D Starting Job spooling tools...=0D=0D [.[32m OK .[0m] Started Job spooling tools.=0D=0D Starting Wait for Plymouth Boot Screen to Quit...=0D=0D Starting Terminate Plymouth Boot Screen...=0D=0D Starting Qubes misc post-boot actions...=0D=0D Starting Qubes GUI Agent...=0D=0D =0D=0D Fedora release 21 (Twenty One)=0D Kernel 4.4.0-1.pvops.qubes.x86_64 on an x86_64 (hvc0)=0D =0D netdebug login: [A=08 =08=08 =08root=0D=0D Last login: Fri Jan 22 01:18:08 on =0D [root@netdebug ~]# ping =08=08=08=08=08.[Kip r=0D default via 10.137.2.1 dev eth0 =0D 10.137.2.1 dev eth0 scope link =0D [root@netdebug ~]# ping -f 10.137.2.1=0D PING 10.137.2.1 (10.137.2.1) 56(84) bytes of data.=0D (...)=0D =0D =2E..........[ 111.001689] page:ffffea00004bffc0 count:0 mapcount:0 mappin= g: (null) index:0x0=0D [ 111.001710] flags: 0x3fff8000000000()=0D [ 111.001718] page dumped because: VM_BUG_ON_PAGE(atomic_read(&page->_coun= t) =3D=3D 0)=0D [ 111.001739] ------------[ cut here ]------------=0D [ 111.001746] kernel BUG at include/linux/mm.h:342!=0D [ 111.001753] invalid opcode: 0000 [#1] SMP =0D [ 111.001761] Modules linked in:=0D [ 111.001769] CPU: 1 PID: 23 Comm: xenwatch Not tainted 4.4.0-1.pvops.qube= s.x86_64 #20=0D [ 111.001780] task: ffff880012c88000 ti: ffff880012c7c000 task.ti: ffff880= 012c7c000=0D [ 111.001790] RIP: e030:[] [] __free_= pages+0x38/0x40=0D [ 111.001807] RSP: e02b:ffff880012c7fc30 EFLAGS: 00010246=0D [ 111.001814] RAX: 0000000000000044 RBX: 000077ff80000000 RCX: 00000000000= 00044=0D [ 111.001822] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880013d= 0de10=0D [ 111.001829] RBP: ffff880012c7fc30 R08: 0000000000000000 R09: 00000000000= 00000=0D [ 111.001837] R10: ffffea0000453f00 R11: ffffffff81863ec0 R12: 00000000000= 00000=0D [ 111.001844] R13: ffff880012eb9c00 R14: ffff880012fff000 R15: 00000000000= 00000=0D [ 111.001857] FS: 00007f84805fe700(0000) GS:ffff880013d00000(0000) knlGS:= 0000000000000000=0D [ 111.001867] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b=0D [ 111.001874] CR2: 000055af04b6d000 CR3: 00000000008e8000 CR4: 00000000000= 02660=0D [ 111.001882] Stack:=0D [ 111.001887] ffff880012c7fc50 ffffffff81174a9c 000000000000000e 00000000= 00000004=0D [ 111.001900] ffff880012c7fc80 ffffffff813f709e ffff87ffffffffff ffff8800= 11528d68=0D [ 111.001912] ffff880012eb9c00 ffff880011528000 ffff880012c7fce8 ffffffff= 814b58c2=0D [ 111.001924] Call Trace:=0D [ 111.001932] [] free_pages+0x5c/0x70=0D [ 111.001942] [] gnttab_end_foreign_access+0x12e/0x160= =0D [ 111.001952] [] xennet_disconnect_backend.isra.26+0x16= 2/0x3b0=0D [ 111.001963] [] xennet_remove+0x31/0x80=0D [ 111.001971] [] xenbus_dev_remove+0x55/0xb0=0D [ 111.001980] [] __device_release_driver+0x96/0x130=0D [ 111.002639] [] device_release_driver+0x23/0x30=0D [ 111.002639] [] bus_remove_device+0x101/0x170=0D [ 111.002639] [] device_del+0x139/0x270=0D [ 111.002639] [] ? unregister_xenbus_watch+0x1d0/0x1d0= =0D [ 111.002639] [] device_unregister+0x1e/0x60=0D [ 111.002639] [] xenbus_dev_changed+0xa3/0x1e0=0D [ 111.002639] [] ? _raw_spin_lock_irqsave+0x1b/0x40=0D [ 111.002639] [] ? unregister_xenbus_watch+0x1d0/0x1d0= =0D [ 111.002639] [] frontend_changed+0x25/0x50=0D [ 111.002639] [] xenwatch_thread+0x87/0x140=0D [ 111.002639] [] ? wait_woken+0x80/0x80=0D [ 111.002639] [] kthread+0xd8/0xf0=0D [ 111.002639] [] ? kthread_park+0x60/0x60=0D [ 111.002639] [] ret_from_fork+0x3f/0x70=0D [ 111.002639] [] ? kthread_park+0x60/0x60=0D [ 111.002639] Code: c0 74 1c f0 ff 4f 1c 74 02 5d c3 85 f6 74 07 e8 ff f8 = ff ff 5d c3 31 f6 e8 f6 fd ff ff 5d c3 48 c7 c6 20 27 a2 81 e8 28 47 02 00 = <0f> 0b 66 0f 1f 44 00 00 66 66 66 66 90 48 85 ff 75 02 f3 c3 55 =0D [ 111.002639] RIP [] __free_pages+0x38/0x40=0D [ 111.002639] RSP =0D [ 111.003417] ---[ end trace e389324fec932a31 ]---=0D =2E.............................................................=03=0D --- 10.137.2.1 ping statistics ---=0D 27465 packets transmitted, 27392 received, 0% packet loss, time 11156ms=0D rtt min/avg/max/mdev =3D 0.053/0.071/12.598/0.109 ms, pipe 2, ipg/ewma 0.40= 6/0.060 ms=0D --FCF9ydJxlAJsfRTu Content-Type: text/plain; charset=utf-8 Content-Disposition: attachment; filename="netdebug.conf" netdebug 4e22b40c-e06e-46d1-aaaa-2293487b81b9 4000 400 2 linux /var/lib/qubes/vm-kernels/4.4-debug/vmlinuz /var/lib/qubes/vm-kernels/4.4-debug/initramfs root=/dev/mapper/dmroot ro nomodeset console=hvc0 rd_NO_PLYMOUTH 3 rd.break destroy destroy destroy