2.6.21-rc5-mm4

2.6.21-rc5-mm4

[Andrew Morton]

ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/

- The oops in git-net.patch has been fixed, so that tree has been restored. 
  It is huge.

- Added the device-mapper development tree to the -mm lineup (Alasdair
  Kergon).  It is a quilt tree, living at
  ftp://ftp.kernel.org/pub/linux/kernel/people/agk/patches/2.6/editing/.

- Added davidel's signalfd stuff.



Boilerplate:

- See the `hot-fixes' directory for any important updates to this patchset.

- To fetch an -mm tree using git, use (for example)

  git-fetch git://git.kernel.org/pub/scm/linux/kernel/git/smurf/linux-trees.git tag v2.6.16-rc2-mm1
  git-checkout -b local-v2.6.16-rc2-mm1 v2.6.16-rc2-mm1

- -mm kernel commit activity can be reviewed by subscribing to the
  mm-commits mailing list.

        echo "subscribe mm-commits" | mail majordomo@vger.kernel.org

- If you hit a bug in -mm and it is not obvious which patch caused it, it is
  most valuable if you can perform a bisection search to identify which patch
  introduced the bug.  Instructions for this process are at

        http://www.zip.com.au/~akpm/linux/patches/stuff/bisecting-mm-trees.txt

  But beware that this process takes some time (around ten rebuilds and
  reboots), so consider reporting the bug first and if we cannot immediately
  identify the faulty patch, then perform the bisection search.

- When reporting bugs, please try to Cc: the relevant maintainer and mailing
  list on any email.

- When reporting bugs in this kernel via email, please also rewrite the
  email Subject: in some manner to reflect the nature of the bug.  Some
  developers filter by Subject: when looking for messages to read.

- Occasional snapshots of the -mm lineup are uploaded to
  ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/mm/ and are announced on
  the mm-commits list.




Changes since 2.6.21-rc5-mm3:


 origin.patch
 git-acpi.patch
 git-alsa.patch
 git-agpgart.patch
 git-arm.patch
 git-avr32.patch
 git-cifs.patch
 git-cpufreq.patch
 git-powerpc.patch
 git-drm.patch
 git-dvb.patch
 git-gfs2-nmw.patch
 git-hid.patch
 git-ia64.patch
 git-ieee1394.patch
 git-infiniband.patch
 git-input.patch
 git-kbuild.patch
 git-kvm.patch
 git-leds.patch
 git-libata-all.patch
 git-md-accel.patch
 git-md-accel-fix.patch
 git-mips.patch
 git-mmc.patch
 git-mtd.patch
 git-ubi.patch
 git-netdev-all.patch
 git-e1000.patch
 git-net.patch
 git-ioat.patch
 git-ocfs2.patch
 git-parisc.patch
 git-r8169.patch
 git-selinux.patch
 git-pciseg.patch
 git-s390.patch
 git-scsi-misc.patch
 git-block.patch
 git-unionfs.patch
 git-watchdog.patch
 git-wireless.patch
 git-ipwireless_cs.patch
 git-cryptodev.patch
 git-gccbug.patch

 git trees.

-proc-fix-linkage-with-config_sysctl=y-config_proc_sysctl=n.patch
-uml-fix-unreasonably-long-udelay.patch
-fix-firmware-sample-code.patch
-jdelvare-i2c-i2c-algo-bit-document-udelay.patch
-pcmcia-allow-pcmcia-scsi-drivers-to-be-built-into-the.patch
-gregkh-pci-pci-set-pci-bfsort-for-poweredge-r900.patch
-fix-gregkh-pci-pci-piggy-bus.patch
-drivers-scsi-dpt_i2oc-remove-dead-code.patch
-scsi-whitespace-cleanup-in-the-dpt-driver.patch
-drivers-scsi-aic7xxx-make-functions-static.patch
-remove-some-unused-scsi-related-kernel-config-variables.patch
-drivers-scsi-aacraid-cleanups.patch
-make-mptspi_target_destroy-static.patch
-qla2xxx-remove-duplicate-pci_disable_device-call.patch
-gregkh-usb-usb-gtcoc-fix-a-use-before-check.patch
-gregkh-usb-usb-ati_remote2-add-channel-support.patch
-usb-serial-whiteheat-convert-to-generic-boolean.patch
-x86_64-mm-dont-probe-for-ddc-on-vbe1_2.patch
-x86_64-mm-remove-hardcoding-of-hard_smp_processor_id-on-up-systems.patch
-drivers-mfd-sm501c-fix-an-off-by-one.patch

 Merged into mainline or a subsystem tree.

+md-avoid-a-deadlock-when-removing-a-device-from-an-md-array-via-sysfs.patch
+md-avoid-a-deadlock-when-removing-a-device-from-an-md-array-via-sysfs-fix.patch
+revert-driver-core-do-not-wait-unnecessarily-in-driver_unregister.patch

 2.6.21 queue.

+vmi-paravirt-ops-bugfix-for-2621.patch

 Might be 2.6.21 queue.

+drivers-acpi-kconfig-formulation-fixpatch.patch

 ACPI fixlet

+pata_platform-for-arm-riscpc.patch

 ARM/pata fix

+cifs-use-mutexdiff.patch

 CIFS cleanup

+agk-dm-dm-merge-max_hw_sector.patch
+agk-dm-dm-raid1-one-kmirrord-per-mirror.patch
+agk-dm-dm-crypt-disable-barriers.patch
+agk-dm-dm-crypt-add-null-iv.patch
+agk-dm-dm-mpath-log-device-name.patch
+agk-dm-dm-allow-offline-devices.patch
+agk-dm-dm-log-fault-detection.patch
+agk-dm-dm-log-report-fault-status.patch
+agk-dm-dm-raid1-add-handle_errors-feature-flag.patch
+agk-dm-dm-io-delay-dec_count.patch
+agk-dm-dm-io-prepare-for-new-interface.patch
+agk-dm-dm-io-new-interface.patch
+agk-dm-dm-kcopyd-update-dm-io-interface.patch
+agk-dm-dm-exception-store-update-dm-io-interface.patch
+agk-dm-dm-log-update-dm-io-interface.patch
+agk-dm-dm-raid1-update-dm-io-interface.patch
+agk-dm-dm-io-remove-old-interface.patch

 Device mapper development tree.

+uevent-use-add_uevent_var-instead-of-open-coding-it.patch

 uevent cleanup

+jdelvare-i2c-i2c-algo-bit-50-50-duty-cycle-4.patch
+jdelvare-i2c-i2c-algo-bit-improve-debug-4.patch

 I2C tree updates

+hid-add-two-led-codes-to-hid-input-mapping.patch

 HID fixlet.

+convert-input-core-to-struct-device.patch
+add-locking-to-evdev.patch

 Input updates

-kbuild-override-build-timestamp.patch

 Dropped, I think.

+testing-patch-for-ali-pata-fixes-hopefully-for-the-problems-with-atapi-dma-fix.patch

 Fix
 testing-patch-for-ali-pata-fixes-hopefully-for-the-problems-with-atapi-dma.patch

+git-mmc-versus-uevent-use-add_uevent_var-instead-of-open-coding-it.patch

 Fix git-mmc for the uevent changes in -mm.

+drivers-net-vioc-possible-cleanups.patch

 Netdev fixes

+2621-rc5-mm3-fix-e1000-compilation.patch

 Fix git-e1000.patch compilation.

+make-drivers-net-qla3xxxcphy_devices-static.patch
+forcedeth-improve-napi-logic.patch

 netdev fixes

+git-net-fixup.patch
+git-net-vs-git-netdev-all.patch

 Fix things due to git-net.patch.

+round_up-macro-cleanup-in-drivers-parisc.patch

 parisc cleanup

+gregkh-pci-pci-flush-msi-x-table-writes.patch
+gregkh-pci-pci-remove-the-broken-pci_multithread_probe-option.patch

 PCI tree updates

+fix-gregkh-pci-pci-remove-the-broken-pci_multithread_probe-option.patch

 Remove now-dead code.

+pci-define-pci_request-release_regions-for-config_pci=n.patch
+round_up-macro-cleanup-in-drivers-pci.patch

 PCI updates

+gregkh-usb-usb-nikon-d80-unusual-device-patch.patch
+gregkh-usb-usb-cypress_cy7c63-race-disconnect-sysfs-read-write-leading-to-following-null-pointer.patch
+gregkh-usb-usb-fix-catc-error-handling.patch
+gregkh-usb-usb-fix-error-handling-in-kl5kusb.patch
+gregkh-usb-usb-fix-error-handling-for-mct_u232.patch
+gregkh-usb-usb-fix-omninet-write-vs-close-race.patch
+gregkh-usb-usb-whiteheat-convert-to-generic-boolean.patch
+gregkh-usb-usb-add-last_busy-field-for-use-in-autosuspend.patch

 USB tree updates

+git-wireless-debug-build-fixes.patch

 Fix git-wireless.patch

+git-wireless-vs-git-net.patch
+git-wireless-vs-git-net-2.patch

 Fix clashes between git-wireless and git-net.

+x86_64-mm-x86-64-eflags-header.patch

 x86_64 tree update

+fix-x86_64-mm-relocatable-kernel-support.patch

 Fix patch in x86_64 tree

+cxgb3-safeguard-tcam-size-usage.patch
+cxgb3-detect-nic-only-adapters.patch
+cxgb3-tighten-xgmac-workaround.patch
+cxgb3-firwmare-update.patch
+fix-hotplug-for-legacy-platform-drivers.patch
+update-documentation-driver-model-platformtxt.patch
+forcedeth-work-around-null-skb-dereference-crash.patch
+fix-scsi_send_eh_cmnd-scatterlist-handling.patch

 These are patches which I think should be in 2.6.21, but vie the relevant
 subsystem maintainers.

+slab-mark-set_up_list3s-__init.patch
+slab-mention-slab-name-when-listing-corrupt-objects.patch
+slub-core.patch
+slub-add-slabinfo-tool.patch

 Memory management updates

+blackfin-arch-fix-bug-interrupt-setup-problem-request_irq.patch
+blackfin-arch-pnav-and-bluetechnix-cm-bf537-use-the-mac.patch

 Blackfin updates

+uml-fix-umid-in-xterm-titles.patch
+uml-speed-up-exec.patch
+uml-no-locking-needed-in-tlsc.patch
+uml-tidy-processc.patch
+uml-remove-page_size.patch
+uml-kernel_thread-shouldnt-panic.patch

 UML updates

+provide-dummy-devm_ioport_-if-has_ioport.patch

 Needed for ARM pata support.

-add-an-anonymous-inode-source.patch
-add-an-anonymous-inode-source-tidy.patch
-epoll-use-anonymous.patch

 Dropped - new versions were merged.

+add-file-position-info-to-proc-fix.patch

 Fix add-file-position-info-to-proc.patch

+move-die-notifier-handling-to-common-code-fix-vmalloc_sync_all.patch

 Fix move-die-notifier-handling-to-common-code.patch some more.

+fix-82875-pci-setup.patch
+make-dev-port-conditional-on-config-symbol.patch
+clean-up-elf-note-generation.patch
+remove-artificial-software-max_loop-limit.patch
+kdump-kexec-calculate-note-size-at-compile-time.patch
+kdump-kexec-calculate-note-size-at-compile-time-update.patch
+fix-kevents-childs-priority-greediness.patch
+fix-kevents-childs-priority-greediness-fix.patch
+display-all-possible-partitions-when-the-root-filesystem-failed-to-mount.patch
+display-all-possible-partitions-when-the-root-filesystem-failed-to-mount-fix.patch
+enhance-initcall_debug-measure-latency.patch
+kprobes-print-details-of-kretprobe-on-assertion-failure.patch
+reregister_binfmt-returns-with-ebusy.patch

 Misc

+sched-implement-staircase-deadline-cpu-scheduler-staircase-improvements.patch

 Staircase scheduler update

+rtc-cmos-wakeup-interface.patch
+acpi-wakeup-hooks-for-rtc-cmos.patch

 RTC stuff.

+sys_futex64-allows-64bit-futexes-workaround-for-uml.patch

 Fix sys_futex64-allows-64bit-futexes.patch some more.

+clocksource-arm-initialize-list-value.patch
+clocksource-parisc-initialize-list-value.patch
+clocksource-avr32-initialize-list-value.patch
+clocksource-mips-initialize-list-value.patch
+clocksource-i386-initialize-list-value.patch
+clocksource-x86_64-initialize-list-value.patch
+clocksource-s390-initialize-list-value.patch
+clocksource-driver-initialize-list-value.patch
+clocksource-refactor-duplicate-registration-checking.patch

 clocksource cleanup

+kprobes-use-hlist_for_each_entry.patch
+kprobes-codingstyle-cleanups.patch
+kprobes-kretprobes-simplifcations.patch

 kprobes cleanups

+utrace-uml-make-uml-compile-with-utrace-enabled.patch

 Fix utrace on UML

+signal-timer-event-fds-v9-anonymous-inode-source.patch
+signal-timer-event-fds-v9-signalfd-core.patch
+signal-timer-event-fds-v9-signalfd-core-fix.patch
+signal-timer-event-fds-v9-signalfd-wire-up-i386-arch.patch
+signal-timer-event-fds-v9-signalfd-wire-up-x86_64-arch.patch
+signal-timer-event-fds-v9-signalfd-compat-code.patch
+signal-timer-event-fds-v9-timerfd-core.patch
+signal-timer-event-fds-v9-timerfd-wire-up-i386-arch.patch
+signal-timer-event-fds-v9-timerfd-wire-up-x86_64-arch.patch
+signal-timer-event-fds-v9-timerfd-compat-code.patch
+signal-timer-event-fds-v9-eventfd-core.patch
+signal-timer-event-fds-v9-eventfd-core-fix.patch
+signal-timer-event-fds-v9-eventfd-wire-up-i386-arch.patch
+signal-timer-event-fds-v9-eventfd-wire-up-x86_64-arch.patch
+signal-timer-event-fds-v9-kaio-eventfd-support-example.patch
+epoll-use-anonymous-inodes.patch

 New types of fds for delivering signals, timers and misc events.

+revoke-wire-up-i386-system-calls-x86_64-fix.patch

 Fix revoke for i386-on-x86_64

+nvidiafb-fix-reversed-ddc-port.patch
+vt-allow-for-the-palette-to-be-exposed-and-changed-via-sysfs.patch
+vt-add-color-support-to-the-underline-and-italic-attributes.patch
+vt-add-color-support-to-the-underline-and-italic-attributes-fix.patch
+fbcon-dont-draw-cursor-when-its-disabled.patch
+vt-expose-system-wide-utf-8-default-setting-via-sysfs.patch

 fbdev updates

-mm-only-hrtimers-debug-patch.patch
-mm-only-hrtimers-debug-patch-fix.patch

 This broke.



All 1254 patches:

ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/patch-list

Re: 2.6.21-rc5-mm4

[Jiri Kosina]

On Mon, 2 Apr 2007, Andrew Morton wrote:

> ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/

[proper CCs added]

On boot (e1000 compiled as module)

BUG: unable to handle kernel paging request at virtual address ffffffff
 printing eip:
ffffffff
*pde = 00005067
*pte = 00000000
Oops: 0000 [#1]
SMP
last sysfs file: 
devices/pci0000:00/0000:00:1c.0/0000:02:00.0/subsystem_device
Modules linked in: cpufreq_conservative cpufreq_ondemand cpufreq_userspace 
cpufreq_powersave acpi_cpufreq freq_table button battery ac loop dm_mod 
usbhid fe
CPU:    0
EIP:    0060:[<ffffffff>]    Not tainted VLI
EFLAGS: 00010002   (2.6.21-rc5-mm4-default #3)
EIP is at 0xffffffff
eax: 80080783   ebx: c242c794   ecx: 00000000   edx: c2371ac0
esi: 00000007   edi: f8cc0000   ebp: 0000003c   esp: c3885ddc
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0068
Process ifplugd (pid: 3185, ti=c3884000 task=c3474070 task.ti=c3884000)
Stack: f8c8a5ed 00000060 00000202 00000000 00000000 c2371ac0 f8c8648c 
c242c794
       c2371ac0 c242c000 f8c810b6 00000001 0000000f 00000000 00008002 
c242c794
       c2371ac0 c242c794 f8c9552c 00000001 0000000f c242c580 c242c000 
c242c240
Call Trace:
 [<f8c8a5ed>] e1000_mc_addr_list_update_generic+0x87/0x146 [e1000]
 [<f8c8648c>] e1000_mc_addr_list_update_82571+0x0/0x3c [e1000]
 [<f8c810b6>] e1000_mc_addr_list_update+0x21/0x27 [e1000]
 [<f8c9552c>] e1000_set_multi+0xe4/0x10c [e1000]
 [<c0272d9f>] __dev_mc_upload+0x23/0x27
 [<c0272ecc>] dev_mc_upload+0x27/0x3b
 [<c027113f>] dev_open+0x4d/0x6d
 [<c026f9c3>] dev_change_flags+0x50/0x104
 [<c02ae726>] devinet_ioctl+0x259/0x57c
 [<c01d8645>] copy_to_user+0x3c/0x53
 [<c0265f4b>] sock_ioctl+0x1a4/0x1c7
 [<c0265da7>] sock_ioctl+0x0/0x1c7
 [<c0180617>] do_ioctl+0x1f/0x64
 [<c018088d>] vfs_ioctl+0x231/0x244
 [<c0267898>] sys_socketcall+0x7a/0x261
 [<c01808ec>] sys_ioctl+0x4c/0x68
 [<c0103dc8>] sysenter_past_esp+0x5d/0x99
 =======================
INFO: lockdep is turned off.
Code:  Bad EIP value.
EIP: [<ffffffff>] 0xffffffff SS:ESP 0068:c3885ddc
Kernel panic - not syncing: Fatal exception in interrupt

-- 
Jiri Kosina

Re: 2.6.21-rc5-mm4

[Kok, Auke]

Jiri Kosina wrote:
> On Mon, 2 Apr 2007, Andrew Morton wrote:
> 
>> ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/
> 
> [proper CCs added]
> 
> On boot (e1000 compiled as module)

can you send me a full `lspci -vv`, `dmesg` as well? I've never seen this happen 
but it appears we did something bad.

Also, it would help a lot if you knew what kind of settings your init scripts 
are trying to configure. It seems that something is setting multicast or 
promiscuous mode. Do you happen to know which command is being executed by the 
shell?

Auke


> BUG: unable to handle kernel paging request at virtual address ffffffff
>  printing eip:
> ffffffff
> *pde = 00005067
> *pte = 00000000
> Oops: 0000 [#1]
> SMP
> last sysfs file: 
> devices/pci0000:00/0000:00:1c.0/0000:02:00.0/subsystem_device
> Modules linked in: cpufreq_conservative cpufreq_ondemand cpufreq_userspace 
> cpufreq_powersave acpi_cpufreq freq_table button battery ac loop dm_mod 
> usbhid fe
> CPU:    0
> EIP:    0060:[<ffffffff>]    Not tainted VLI
> EFLAGS: 00010002   (2.6.21-rc5-mm4-default #3)
> EIP is at 0xffffffff
> eax: 80080783   ebx: c242c794   ecx: 00000000   edx: c2371ac0
> esi: 00000007   edi: f8cc0000   ebp: 0000003c   esp: c3885ddc
> ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0068
> Process ifplugd (pid: 3185, ti=c3884000 task=c3474070 task.ti=c3884000)
> Stack: f8c8a5ed 00000060 00000202 00000000 00000000 c2371ac0 f8c8648c 
> c242c794
>        c2371ac0 c242c000 f8c810b6 00000001 0000000f 00000000 00008002 
> c242c794
>        c2371ac0 c242c794 f8c9552c 00000001 0000000f c242c580 c242c000 
> c242c240
> Call Trace:
>  [<f8c8a5ed>] e1000_mc_addr_list_update_generic+0x87/0x146 [e1000]
>  [<f8c8648c>] e1000_mc_addr_list_update_82571+0x0/0x3c [e1000]
>  [<f8c810b6>] e1000_mc_addr_list_update+0x21/0x27 [e1000]
>  [<f8c9552c>] e1000_set_multi+0xe4/0x10c [e1000]
>  [<c0272d9f>] __dev_mc_upload+0x23/0x27
>  [<c0272ecc>] dev_mc_upload+0x27/0x3b
>  [<c027113f>] dev_open+0x4d/0x6d
>  [<c026f9c3>] dev_change_flags+0x50/0x104
>  [<c02ae726>] devinet_ioctl+0x259/0x57c
>  [<c01d8645>] copy_to_user+0x3c/0x53
>  [<c0265f4b>] sock_ioctl+0x1a4/0x1c7
>  [<c0265da7>] sock_ioctl+0x0/0x1c7
>  [<c0180617>] do_ioctl+0x1f/0x64
>  [<c018088d>] vfs_ioctl+0x231/0x244
>  [<c0267898>] sys_socketcall+0x7a/0x261
>  [<c01808ec>] sys_ioctl+0x4c/0x68
>  [<c0103dc8>] sysenter_past_esp+0x5d/0x99
>  =======================
> INFO: lockdep is turned off.
> Code:  Bad EIP value.
> EIP: [<ffffffff>] 0xffffffff SS:ESP 0068:c3885ddc
> Kernel panic - not syncing: Fatal exception in interrupt
> 

Re: 2.6.21-rc5-mm4

[Jiri Kosina]

On Tue, 3 Apr 2007, Kok, Auke wrote:

> Also, it would help a lot if you knew what kind of settings your init 
> scripts are trying to configure. It seems that something is setting 
> multicast or promiscuous mode. Do you happen to know which command is 
> being executed by the shell?

It's 100% reproducible by running dhclient on the interface.

The kernel also panics (with different stacktrace but the same wrong 
address it tries to dereference - 0xffffffff) when I do this:

(none):/# modprobe e1000
Intel(R) PRO/1000 Network Driver - version 7.4.35-k2-NAPI
Copyright (c) 1999-2007 Intel Corporation.
ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 16 (level, low) -> IRQ 16
e1000: 0000:02:00.0: e1000_probe: (PCI Express:2.5Gb/s:Width x1) 
00:0e:0c:72:b8:ad
e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
(none):/# ifconfig eth0 up
BUG: unable to handle kernel paging request at virtual address ffffffff
 printing eip:
ffffffff
*pde = 00005067
*pte = 00000000
Oops: 0000 [#1]
SMP
last sysfs file: power/resume
Modules linked in: e1000 ext3 mbcache jbd edd sg fan generic ata_piix 
libata thermal processor sd_mod scsi_mod ide_disk ide_core
CPU:    0
EIP:    0060:[<ffffffff>]    Not tainted VLI
EFLAGS: 00010046   (2.6.21-rc5-mm4-default #3)
EIP is at 0xffffffff
eax: 00000000   ebx: 00000000   ecx: 00000000   edx: 00000000
esi: 00000000   edi: c0462008   ebp: 00000080   esp: c0463f94
ds: 007b   es: 007b   fs: 00d8  gs: 0000  ss: 0068
Process swapper (pid: 0, ti=c0462000 task=c038d340 task.ti=c0462000)
Stack: c0102297 00000060 00000246 00000000 c049f5a4 c1d095a4 c01023fa c0363b12
       20070126 c1d11000 c0467ae7 c034a48a c04671b9 00000000 00000000 00000000
       00000000 00010000 00000000 00000000 00000000 c04887c0 00020800 00099800
Call Trace:
 [<c0102297>] mwait_idle_with_hints+0x3b/0x3f
 [<c01023fa>] cpu_idle+0xa6/0xbf
 [<c0467ae7>] start_kernel+0x459/0x461
 [<c04671b9>] unknown_bootoption+0x0/0x202
 =======================
INFO: lockdep is turned off.
Code:  Bad EIP value.
EIP: [<ffffffff>] 0xffffffff SS:ESP 0068:c0463f94
Kernel panic - not syncing: Attempted to kill the idle task!


Seeing "mwait" there, it seemed like ACPI might be involved. Booting with 
acpi=off didn't improve the situation though.

I just pulled your e1000 git tree into vanilla 2.6.21-rc5 ytrr and I can't 
reproduce the problem, so it might be caused by something else.

Will try to narrow down what is causing it.

-- 
Jiri Kosina

Re: 2.6.21-rc5-mm4

[Kok, Auke]

Jiri Kosina wrote:
> On Tue, 3 Apr 2007, Kok, Auke wrote:
> 
>> Also, it would help a lot if you knew what kind of settings your init 
>> scripts are trying to configure. It seems that something is setting 
>> multicast or promiscuous mode. Do you happen to know which command is 
>> being executed by the shell?
> 
> It's 100% reproducible by running dhclient on the interface.
> 
> The kernel also panics (with different stacktrace but the same wrong 
> address it tries to dereference - 0xffffffff) when I do this:
> 
> (none):/# modprobe e1000
> Intel(R) PRO/1000 Network Driver - version 7.4.35-k2-NAPI
> Copyright (c) 1999-2007 Intel Corporation.
> ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 16 (level, low) -> IRQ 16
> e1000: 0000:02:00.0: e1000_probe: (PCI Express:2.5Gb/s:Width x1) 
> 00:0e:0c:72:b8:ad
> e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
> (none):/# ifconfig eth0 up
> BUG: unable to handle kernel paging request at virtual address ffffffff
>  printing eip:
> ffffffff
> *pde = 00005067
> *pte = 00000000
> Oops: 0000 [#1]
> SMP
> last sysfs file: power/resume
> Modules linked in: e1000 ext3 mbcache jbd edd sg fan generic ata_piix 
> libata thermal processor sd_mod scsi_mod ide_disk ide_core
> CPU:    0
> EIP:    0060:[<ffffffff>]    Not tainted VLI
> EFLAGS: 00010046   (2.6.21-rc5-mm4-default #3)
> EIP is at 0xffffffff
> eax: 00000000   ebx: 00000000   ecx: 00000000   edx: 00000000
> esi: 00000000   edi: c0462008   ebp: 00000080   esp: c0463f94
> ds: 007b   es: 007b   fs: 00d8  gs: 0000  ss: 0068
> Process swapper (pid: 0, ti=c0462000 task=c038d340 task.ti=c0462000)
> Stack: c0102297 00000060 00000246 00000000 c049f5a4 c1d095a4 c01023fa c0363b12
>        20070126 c1d11000 c0467ae7 c034a48a c04671b9 00000000 00000000 00000000
>        00000000 00010000 00000000 00000000 00000000 c04887c0 00020800 00099800
> Call Trace:
>  [<c0102297>] mwait_idle_with_hints+0x3b/0x3f
>  [<c01023fa>] cpu_idle+0xa6/0xbf
>  [<c0467ae7>] start_kernel+0x459/0x461
>  [<c04671b9>] unknown_bootoption+0x0/0x202
>  =======================
> INFO: lockdep is turned off.
> Code:  Bad EIP value.
> EIP: [<ffffffff>] 0xffffffff SS:ESP 0068:c0463f94
> Kernel panic - not syncing: Attempted to kill the idle task!
> 
> 
> Seeing "mwait" there, it seemed like ACPI might be involved. Booting with 
> acpi=off didn't improve the situation though.
> 
> I just pulled your e1000 git tree into vanilla 2.6.21-rc5 ytrr and I can't 
> reproduce the problem, so it might be caused by something else.

we're also having problems reproducing it on that same combination (2.6.21-rc4 + 
my tree), so it points to something in -mm. Since your trace is completely 
different right now it looks like something else is fuzzing it up.

Since the e1000 changes are in rc5-mm3 as well, that might help to narrow it 
down quickly.

Auke

Re: 2.6.21-rc5-mm4

[Jiri Kosina]

On Tue, 3 Apr 2007, Kok, Auke wrote:

> we're also having problems reproducing it on that same combination 
> (2.6.21-rc4 + my tree), so it points to something in -mm. Since your 
> trace is completely different right now it looks like something else is 
> fuzzing it up.
> Since the e1000 changes are in rc5-mm3 as well, that might help to 
> narrow it down quickly.

I don't know (yet) whether rc5-mm3 was OK in this respect, I didn't boot 
it on this machine. I only know that both rc5 and rc5 + e1000 tree are OK, 
but rc5-mm4 panics on ifconfig/dhclient on e1000 card immediately on my 
system.

I will start bisection when I get back to the respective machine 
(tomorrow) and will let you know.

-- 
Jiri Kosina

Re: 2.6.21-rc5-mm4

[Jiri Kosina]

On Tue, 3 Apr 2007, Jiri Kosina wrote:

> > we're also having problems reproducing it on that same combination 
> > (2.6.21-rc4 + my tree), so it points to something in -mm. Since your 
> > trace is completely different right now it looks like something else 
> > is fuzzing it up. Since the e1000 changes are in rc5-mm3 as well, that 
> > might help to narrow it down quickly.
> I don't know (yet) whether rc5-mm3 was OK in this respect, I didn't boot 
> it on this machine. I only know that both rc5 and rc5 + e1000 tree are 
> OK, but rc5-mm4 panics on ifconfig/dhclient on e1000 card immediately on 
> my system.
> I will start bisection when I get back to the respective machine 
> (tomorrow) and will let you know.

And the bisection winner is

	i386-irq-kill-nr_irq_vectors-and-increase-nr_irqs.patch

I don't immediately see how it could be causing it, so adding CCs which 
are listed in the patch.

Original description of the symptoms at http://lkml.org/lkml/2007/4/3/90

-- 
Jiri Kosina

Re: 2.6.21-rc5-mm4

[Eric W. Biederman]

Jiri Kosina <jikos@jikos.cz> writes:

> On Tue, 3 Apr 2007, Jiri Kosina wrote:
>
>> > we're also having problems reproducing it on that same combination 
>> > (2.6.21-rc4 + my tree), so it points to something in -mm. Since your 
>> > trace is completely different right now it looks like something else 
>> > is fuzzing it up. Since the e1000 changes are in rc5-mm3 as well, that 
>> > might help to narrow it down quickly.
>> I don't know (yet) whether rc5-mm3 was OK in this respect, I didn't boot 
>> it on this machine. I only know that both rc5 and rc5 + e1000 tree are 
>> OK, but rc5-mm4 panics on ifconfig/dhclient on e1000 card immediately on 
>> my system.
>> I will start bisection when I get back to the respective machine 
>> (tomorrow) and will let you know.
>
> And the bisection winner is
>
> 	i386-irq-kill-nr_irq_vectors-and-increase-nr_irqs.patch
>
> I don't immediately see how it could be causing it, so adding CCs which 
> are listed in the patch.

Weird.  I will have to look at that in a little more detail.

Do you know if this problem happens on x86_64?
What does your .config look like?
What does /proc/interrupts look like?
What kind of hardware you running this kernel on?
Can anyone else reproduce this?

The oops clearly shows something using -1 and calling that as an
address I don't know why, but I'm guessing I have triggered a memory
stomp somewhere.  I think this is the first time I have seen a small
negative number causing a NULL pointer dereference.

That patch looks innocuous enough that either:
- I just missed changing something I should have.
- Your configuration has an increase in NR_IRQS and that triggered
  something.
- The patch simply permuted things so a memory stomp now happens
  on the e1000 data structures instead of somewhere else.
- Something doesn't like large irq numbers.

This work is essentially a backport from x86_64 so if your hardware
is 64bit capable testing that should be a fairly easy test, and be
able to rule out large irq numbers as the culprit.

Until I get a good look at -mm I'm going to have a hard time guessing.
But a roving memory stomp is my best guess.


Eric

Re: 2.6.21-rc5-mm4

[Jiri Kosina]

On Wed, 4 Apr 2007, Eric W. Biederman wrote:

> > And the bisection winner is
> >
> > 	i386-irq-kill-nr_irq_vectors-and-increase-nr_irqs.patch
> >
> > I don't immediately see how it could be causing it, so adding CCs which 
> > are listed in the patch.
> Weird.  I will have to look at that in a little more detail.
> Do you know if this problem happens on x86_64? What does your .config 
> look like? What does /proc/interrupts look like? What kind of hardware 
> you running this kernel on? Can anyone else reproduce this?
> The oops clearly shows something using -1 and calling that as an
> address I don't know why, but I'm guessing I have triggered a memory
> stomp somewhere.  I think this is the first time I have seen a small
> negative number causing a NULL pointer dereference.
> That patch looks innocuous enough that either:
> - I just missed changing something I should have.
> - Your configuration has an increase in NR_IRQS and that triggered
>   something.
> - The patch simply permuted things so a memory stomp now happens
>   on the e1000 data structures instead of somewhere else.
> - Something doesn't like large irq numbers.
> This work is essentially a backport from x86_64 so if your hardware
> is 64bit capable testing that should be a fairly easy test, and be
> able to rule out large irq numbers as the culprit.
> Until I get a good look at -mm I'm going to have a hard time guessing.
> But a roving memory stomp is my best guess.

Hi Eric,

after struggling with this issue for some time, I think that it's just 
some incosistent usage of NR_IRQS throughout the source probably due to 
some include hell. I really don't understand the how the mach-*/ includes 
are supposed to work.

I found out (by disassembling resulting vmlinux binaries) that in 
arch/i386/kernel/entry.S, the loop in irq_entries_start does too little 
iterations compared to NR_IRQS value as seen in for example io_apic.c

The super-stupid proof-patch below fixes the panic on my system. It's just 
to demonstrate that the i386 includes really need fixing to be consistent 
somehow.

diff --git a/arch/i386/kernel/entry.S b/arch/i386/kernel/entry.S
index 976438c..b20dc07 100644
--- a/arch/i386/kernel/entry.S
+++ b/arch/i386/kernel/entry.S
@@ -53,6 +53,8 @@
 #include <asm/dwarf2.h>
 #include "irq_vectors.h"
 
+#define NR_IRQS 4096
+
 /*
  * We use macros for low-level operations which need to be overridden
  * for paravirtualization.  The following will never clobber any registers:

-- 
Jiri Kosina

Re: 2.6.21-rc5-mm4

[Eric W. Biederman]

Jiri Kosina <jikos@jikos.cz> writes:

> Hi Eric,
>
> after struggling with this issue for some time, I think that it's just 
> some incosistent usage of NR_IRQS throughout the source probably due to 
> some include hell. I really don't understand the how the mach-*/ includes 
> are supposed to work.
>
> I found out (by disassembling resulting vmlinux binaries) that in 
> arch/i386/kernel/entry.S, the loop in irq_entries_start does too little 
> iterations compared to NR_IRQS value as seen in for example io_apic.c
>
> The super-stupid proof-patch below fixes the panic on my system. It's just 
> to demonstrate that the i386 includes really need fixing to be consistent 
> somehow.

Thanks, and that would do it, it  makes sense why it was the irq patch
that caused problems.  I had forgotten about the number of stubs issue.

I had to clean that up on x86_64 as well and it probably makes most sense
to put that cleanup as well, so we have a small fixed number of stubs
which would make the includes not matter.

Bleh.  Hopefully soon.

Eric

> diff --git a/arch/i386/kernel/entry.S b/arch/i386/kernel/entry.S
> index 976438c..b20dc07 100644
> --- a/arch/i386/kernel/entry.S
> +++ b/arch/i386/kernel/entry.S
> @@ -53,6 +53,8 @@
>  #include <asm/dwarf2.h>
>  #include "irq_vectors.h"
>  
> +#define NR_IRQS 4096
> +
>  /*
>   * We use macros for low-level operations which need to be overridden
>   * for paravirtualization.  The following will never clobber any registers:
>
> -- 
> Jiri Kosina

Re: 2.6.21-rc5-mm4

[Michal Piotrowski]

Andrew Morton napisał(a):
> ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/
> 

Looks like a scheduler problem.

BUG: unable to handle kernel paging request at virtual address 0743c798
 printing eip:
c011d840
*pde = 00000000
Oops: 0000 [#1]
PREEMPT SMP 
last sysfs file: devices/platform/w83627hf.656/temp2_input
Modules linked in: ipt_MASQUERADE iptable_nat nf_nat nfsd exportfs lockd nfs_acl autofs4 sunrpc af_packet nf_conntrack_netbios_ns ipt_REJECT nf_conntrack_ipv4 xt_state nf_conntrack nfnetlink iptable_filter ip_tables ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables ipv6 binfmt_misc thermal processor fan container nvram snd_intel8x0 snd_ac97_codec ac97_bus snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm evdev snd_timer snd soundcore snd_page_alloc intel_agp agpgart i2c_i801 ide_cd cdrom rtc unix
CPU:    0
EIP:    0060:[<c011d840>]    Not tainted VLI

 l *0xc011d840
0xc011d840 is in recalc_task_prio (/mnt/md0/devel/linux-mm/kernel/sched.c:720).
715              * a queued better static_prio's prio_level.
716              */
717             if (p->static_prio < rq->best_static_prio) {
718                     search_prio = MAX_RT_PRIO;
719                     if (likely(p->policy != SCHED_BATCH))
720                             rq->best_static_prio = p->static_prio;
721             } else if (p->static_prio == rq->best_static_prio)
722                     search_prio = rq->prio_level[uprio];
723             else {
724                     search_prio = max(rq->prio_level[uprio],


EFLAGS: 00210006   (2.6.21-rc5-mm4 #13)
EIP is at recalc_task_prio+0xfa/0x19e
eax: 1000008b   ebx: 10000027   ecx: 0000008b   edx: c743bda0
esi: c985a070   edi: c743bda0   ebp: ce47fec0   esp: ce47fe98
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0068
Process operapluginwrap (pid: 3560, ti=ce47e000 task=ccffd510 task.ti=ce47e000)
Stack: 00000000 c743bda0 c743c27c c011df36 c01240ed c04eac14 ce47fed0 c985a070 
       c743bda0 c985a070 ce47fed0 c011da29 00000001 c985a070 ce47ff00 c011f327 
       00000000 c052cda0 ce47ff00 c743bda0 8fbe3987 0000002d c985a070 003d0f00 
Call Trace:
 [<c011da29>] enqueue_task+0x31/0x7d
 [<c011f327>] activate_task+0xba/0xce
 [<c01223f3>] wake_up_new_task+0x65/0x125
 [<c012440d>] do_fork+0x10f/0x181
 [<c010223f>] sys_clone+0x33/0x39
 [<c010424c>] syscall_call+0x7/0xb
 [<4cefd258>] 0x4cefd258
 =======================
INFO: lockdep is turned off.
Code: 83 7e 5c 03 0f 95 c2 b8 74 4c 4a c0 e8 12 e4 0f 00 bf 64 00 00 00 85 c0 74 2f 8b 46 1c 8b 55 dc 89 82 fc 09 00 00 eb 21 8b 7d dc <8b> 94 9f 5c 09 00 00 75 04 89 d7 eb 11 8b 7d dc 8b 84 8f cc 07 
EIP: [<c011d840>] recalc_task_prio+0xfa/0x19e SS:ESP 0068:ce47fe98
BUG: NMI Watchdog detected LOCKUP on CPU1, eip c020cf0f, registers:
Modules linked in: ipt_MASQUERADE iptable_nat nf_nat nfsd exportfs lockd nfs_acl autofs4 sunrpc af_packet nf_conntrack_netbios_ns ipt_REJECT nf_conntrack_ipv4 xt_state nf_conntrack nfnetlink iptable_filter ip_tables ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables ipv6 binfmt_misc thermal processor fan container nvram snd_intel8x0 snd_ac97_codec ac97_bus snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm evdev snd_timer snd soundcore snd_page_alloc intel_agp agpgart i2c_i801 ide_cd cdrom rtc unix
CPU:    1
EIP:    0060:[<c020cf0f>]    Not tainted VLI

l *0xc020cf0f
0xc020cf0f is in delay_tsc (/mnt/md0/devel/linux-mm/arch/i386/lib/delay.c:45).
40      /* TSC based delay: */
41      static void delay_tsc(unsigned long loops)
42      {
43              unsigned long bclock, now;
44
45              rdtscl(bclock);
46              do {
47                      rep_nop();
48                      rdtscl(now);
49              } while ((now-bclock) < loops);


EFLAGS: 00000096   (2.6.21-rc5-mm4 #13)
EIP is at delay_tsc+0x8/0x17
eax: aa2a3a10   ebx: 00000001   ecx: aa2a386c   edx: 00000083
esi: 0262e5e8   edi: 00000001   ebp: ccf1fc80   esp: ccf1fc7c
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0068
Process firefox-bin (pid: 3464, ti=ccf1e000 task=caf66ac0 task.ti=ccf1e000)
Stack: c743bda0 ccf1fc90 c020cedc 00000000 c021add9 ccf1fcc0 c021ade8 00000000 
       00000002 00000001 c011df36 00000000 caf66ac0 a6ca16f0 c743bda0 c743bda0 
       ccf1fd50 ccf1fce0 c034f4be 00000000 00000002 c011df36 00200200 ccea1050 
Call Trace:
 [<c020cedc>] __delay+0xc/0xe
 [<c021ade8>] _raw_spin_lock+0xc3/0x128
 [<c034f4be>] _spin_lock+0x3c/0x42
 [<c011df36>] task_rq_lock+0x36/0x70
 [<c011f355>] try_to_wake_up+0x1a/0x36b
 [<c011f6b4>] default_wake_function+0xe/0x10
 [<c011c773>] __wake_up_common+0x32/0x55
 [<c011d627>] __wake_up+0x31/0x42
 [<c02f847f>] sock_def_readable+0x42/0x70
 [<f8858ed0>] unix_stream_sendmsg+0x22d/0x2e2 [unix]
 [<c02f3f95>] sock_aio_write+0xfb/0x107
 [<c0185cb4>] do_sync_write+0xc5/0x102
 [<c01865aa>] vfs_write+0xe5/0x15a
 [<c0186bc8>] sys_write+0x3d/0x72
 [<c010424c>] syscall_call+0x7/0xb
 [<b7f24410>] 0xb7f24410
 =======================
INFO: lockdep is turned off.
Code: c1 e0 02 64 8b 15 04 00 00 00 c1 e2 08 69 92 88 88 4e c0 fa 00 00 00 f7 e2 8d 42 01 e8 cb ff ff ff c9 c3 55 89 e5 53 89 c3 0f 31 <89> c1 f3 90 0f 31 29 c8 39 d8 72 f6 5b 5d c3 55 89 e5 83 ec 08 
BUG: NMI Watchdog detected LOCKUP on CPU0, eip c020cf15, registers:
Modules linked in: ipt_MASQUERADE iptable_nat nf_nat nfsd exportfs lockd nfs_acl autofs4 sunrpc af_packet nf_conntrack_netbios_ns ipt_REJECT nf_conntrack_ipv4 xt_state nf_conntrack nfnetlink iptable_filter ip_tables ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables ipv6 binfmt_misc thermal processor fan container nvram snd_intel8x0 snd_ac97_codec ac97_bus snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm evdev snd_timer snd soundcore snd_page_alloc intel_agp agpgart i2c_i801 ide_cd cdrom rtc unix
CPU:    0
EIP:    0060:[<c020cf15>]    Not tainted VLI

l *0xc020cf15
0xc020cf15 is in delay_tsc (/mnt/md0/devel/linux-mm/arch/i386/lib/delay.c:49).
44
45              rdtscl(bclock);
46              do {
47                      rep_nop();
48                      rdtscl(now);
49              } while ((now-bclock) < loops);
50      }
51
52      /*
53       * Since we calibrate only once at boot, this


EFLAGS: 00200086   (2.6.21-rc5-mm4 #13)
EIP is at delay_tsc+0xe/0x17
eax: ec55c3c8   ebx: 00000001   ecx: ec55c348   edx: 00000086
esi: 04531454   edi: 00000001   ebp: ce47fc48   esp: ce47fc44
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0068
Process operapluginwrap (pid: 3560, ti=ce47e000 task=ccffd510 task.ti=ce47e000)
Stack: c743bda0 ce47fc58 c020cedc 00000000 c021add9 ce47fc88 c021ade8 00000000 
       00000002 00000001 c011df36 000f4240 000026eb a6ca16f0 c743bda0 c743bda0 
       ce47fd18 ce47fca8 c034f4be 00000000 00000002 c011df36 c011fe36 c011f5b2 
Call Trace:
 [<c020cedc>] __delay+0xc/0xe
 [<c021ade8>] _raw_spin_lock+0xc3/0x128
 [<c034f4be>] _spin_lock+0x3c/0x42
 [<c011df36>] task_rq_lock+0x36/0x70
 [<c011f5bd>] try_to_wake_up+0x282/0x36b
 [<c011f6b4>] default_wake_function+0xe/0x10
 [<c0136fc2>] autoremove_wake_function+0x17/0x3a
 [<c011c773>] __wake_up_common+0x32/0x55
 [<c011d627>] __wake_up+0x31/0x42
 [<c0125457>] wake_up_klogd+0x36/0x38
 [<c020e38a>] bust_spinlocks+0x2a/0x2c
 [<c01058bd>] die+0x16c/0x22c
 [<c011b9ed>] do_page_fault+0x47c/0x55b
 [<c034fdf1>] error_code+0x79/0x80
 [<c011d840>] recalc_task_prio+0xfa/0x19e
 [<c011da29>] enqueue_task+0x31/0x7d
 [<c011f327>] activate_task+0xba/0xce
 [<c01223f3>] wake_up_new_task+0x65/0x125
 [<c012440d>] do_fork+0x10f/0x181
 [<c010223f>] sys_clone+0x33/0x39
 [<c010424c>] syscall_call+0x7/0xb
 [<4cefd258>] 0x4cefd258
 =======================
INFO: lockdep is turned off.
Code: 04 00 00 00 c1 e2 08 69 92 88 88 4e c0 fa 00 00 00 f7 e2 8d 42 01 e8 cb ff ff ff c9 c3 55 89 e5 53 89 c3 0f 31 89 c1 f3 90 0f 31 <29> c8 39 d8 72 f6 5b 5d c3 55 89 e5 83 ec 08 69 c0 1c 43 00 00 

I'll try without this patches

sched-fix-idle-load-balancing-in-softirqd-context.patch
sched-fix-idle-load-balancing-in-softirqd-context-fix.patch
sched-dynticks-idle-load-balancing-v3.patch
speedup-divides-by-cpu_power-in-scheduler.patch
sched-optimize-siblings-status-check-logic-in-wake_idle.patch
#
sched-dont-renice-kernel-threads.patch
sched-remove-sleepavg-from-proc.patch
sched-implement-staircase-deadline-cpu-scheduler.patch
sched-implement-staircase-deadline-cpu-scheduler-misc-fixes.patch
sched-implement-staircase-deadline-cpu-scheduler-staircase-improvements.patch
sched-remove-noninteractive-flag.patch
sched-document-sd-cpu-scheduler.patch
#
sched2-sched-domain-sysctl.patch
#
sched-add-above-background-load-function.patch


http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.21-rc5-mm4/mm-oops
http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.21-rc5-mm4/mm-config

Regards,
Michal

-- 
Michal K. K. Piotrowski
LTG - Linux Testers Group (PL)
(http://www.stardust.webpages.pl/ltg/)
LTG - Linux Testers Group (EN)
(http://www.stardust.webpages.pl/linux_testers_group_en/)

Re: 2.6.21-rc5-mm4

[Michal Piotrowski]

Michal Piotrowski napisał(a):
> Andrew Morton napisał(a):
>> ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/
>>
> 
> Looks like a scheduler problem.
> 
> BUG: unable to handle kernel paging request at virtual address 0743c798
>  printing eip:
> c011d840
> *pde = 00000000
> Oops: 0000 [#1]
> PREEMPT SMP 
> last sysfs file: devices/platform/w83627hf.656/temp2_input
> Modules linked in: ipt_MASQUERADE iptable_nat nf_nat nfsd exportfs lockd nfs_acl autofs4 sunrpc af_packet nf_conntrack_netbios_ns ipt_REJECT nf_conntrack_ipv4 xt_state nf_conntrack nfnetlink iptable_filter ip_tables ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables ipv6 binfmt_misc thermal processor fan container nvram snd_intel8x0 snd_ac97_codec ac97_bus snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm evdev snd_timer snd soundcore snd_page_alloc intel_agp agpgart i2c_i801 ide_cd cdrom rtc unix
> CPU:    0
> EIP:    0060:[<c011d840>]    Not tainted VLI
> 
>  l *0xc011d840
> 0xc011d840 is in recalc_task_prio (/mnt/md0/devel/linux-mm/kernel/sched.c:720).
> 715              * a queued better static_prio's prio_level.
> 716              */
> 717             if (p->static_prio < rq->best_static_prio) {
> 718                     search_prio = MAX_RT_PRIO;
> 719                     if (likely(p->policy != SCHED_BATCH))
> 720                             rq->best_static_prio = p->static_prio;
> 721             } else if (p->static_prio == rq->best_static_prio)
> 722                     search_prio = rq->prio_level[uprio];
> 723             else {
> 724                     search_prio = max(rq->prio_level[uprio],
> 
> 
> EFLAGS: 00210006   (2.6.21-rc5-mm4 #13)
> EIP is at recalc_task_prio+0xfa/0x19e
> eax: 1000008b   ebx: 10000027   ecx: 0000008b   edx: c743bda0
> esi: c985a070   edi: c743bda0   ebp: ce47fec0   esp: ce47fe98
> ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0068
> Process operapluginwrap (pid: 3560, ti=ce47e000 task=ccffd510 task.ti=ce47e000)
> Stack: 00000000 c743bda0 c743c27c c011df36 c01240ed c04eac14 ce47fed0 c985a070 
>        c743bda0 c985a070 ce47fed0 c011da29 00000001 c985a070 ce47ff00 c011f327 
>        00000000 c052cda0 ce47ff00 c743bda0 8fbe3987 0000002d c985a070 003d0f00 
> Call Trace:
>  [<c011da29>] enqueue_task+0x31/0x7d
>  [<c011f327>] activate_task+0xba/0xce
>  [<c01223f3>] wake_up_new_task+0x65/0x125
>  [<c012440d>] do_fork+0x10f/0x181
>  [<c010223f>] sys_clone+0x33/0x39
>  [<c010424c>] syscall_call+0x7/0xb
>  [<4cefd258>] 0x4cefd258
>  =======================
> INFO: lockdep is turned off.
> Code: 83 7e 5c 03 0f 95 c2 b8 74 4c 4a c0 e8 12 e4 0f 00 bf 64 00 00 00 85 c0 74 2f 8b 46 1c 8b 55 dc 89 82 fc 09 00 00 eb 21 8b 7d dc <8b> 94 9f 5c 09 00 00 75 04 89 d7 eb 11 8b 7d dc 8b 84 8f cc 07 
> EIP: [<c011d840>] recalc_task_prio+0xfa/0x19e SS:ESP 0068:ce47fe98
> BUG: NMI Watchdog detected LOCKUP on CPU1, eip c020cf0f, registers:
> Modules linked in: ipt_MASQUERADE iptable_nat nf_nat nfsd exportfs lockd nfs_acl autofs4 sunrpc af_packet nf_conntrack_netbios_ns ipt_REJECT nf_conntrack_ipv4 xt_state nf_conntrack nfnetlink iptable_filter ip_tables ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables ipv6 binfmt_misc thermal processor fan container nvram snd_intel8x0 snd_ac97_codec ac97_bus snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm evdev snd_timer snd soundcore snd_page_alloc intel_agp agpgart i2c_i801 ide_cd cdrom rtc unix
> CPU:    1
> EIP:    0060:[<c020cf0f>]    Not tainted VLI
> 
> l *0xc020cf0f
> 0xc020cf0f is in delay_tsc (/mnt/md0/devel/linux-mm/arch/i386/lib/delay.c:45).
> 40      /* TSC based delay: */
> 41      static void delay_tsc(unsigned long loops)
> 42      {
> 43              unsigned long bclock, now;
> 44
> 45              rdtscl(bclock);
> 46              do {
> 47                      rep_nop();
> 48                      rdtscl(now);
> 49              } while ((now-bclock) < loops);
> 
> 
> EFLAGS: 00000096   (2.6.21-rc5-mm4 #13)
> EIP is at delay_tsc+0x8/0x17
> eax: aa2a3a10   ebx: 00000001   ecx: aa2a386c   edx: 00000083
> esi: 0262e5e8   edi: 00000001   ebp: ccf1fc80   esp: ccf1fc7c
> ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0068
> Process firefox-bin (pid: 3464, ti=ccf1e000 task=caf66ac0 task.ti=ccf1e000)
> Stack: c743bda0 ccf1fc90 c020cedc 00000000 c021add9 ccf1fcc0 c021ade8 00000000 
>        00000002 00000001 c011df36 00000000 caf66ac0 a6ca16f0 c743bda0 c743bda0 
>        ccf1fd50 ccf1fce0 c034f4be 00000000 00000002 c011df36 00200200 ccea1050 
> Call Trace:
>  [<c020cedc>] __delay+0xc/0xe
>  [<c021ade8>] _raw_spin_lock+0xc3/0x128
>  [<c034f4be>] _spin_lock+0x3c/0x42
>  [<c011df36>] task_rq_lock+0x36/0x70
>  [<c011f355>] try_to_wake_up+0x1a/0x36b
>  [<c011f6b4>] default_wake_function+0xe/0x10
>  [<c011c773>] __wake_up_common+0x32/0x55
>  [<c011d627>] __wake_up+0x31/0x42
>  [<c02f847f>] sock_def_readable+0x42/0x70
>  [<f8858ed0>] unix_stream_sendmsg+0x22d/0x2e2 [unix]
>  [<c02f3f95>] sock_aio_write+0xfb/0x107
>  [<c0185cb4>] do_sync_write+0xc5/0x102
>  [<c01865aa>] vfs_write+0xe5/0x15a
>  [<c0186bc8>] sys_write+0x3d/0x72
>  [<c010424c>] syscall_call+0x7/0xb
>  [<b7f24410>] 0xb7f24410
>  =======================
> INFO: lockdep is turned off.
> Code: c1 e0 02 64 8b 15 04 00 00 00 c1 e2 08 69 92 88 88 4e c0 fa 00 00 00 f7 e2 8d 42 01 e8 cb ff ff ff c9 c3 55 89 e5 53 89 c3 0f 31 <89> c1 f3 90 0f 31 29 c8 39 d8 72 f6 5b 5d c3 55 89 e5 83 ec 08 
> BUG: NMI Watchdog detected LOCKUP on CPU0, eip c020cf15, registers:
> Modules linked in: ipt_MASQUERADE iptable_nat nf_nat nfsd exportfs lockd nfs_acl autofs4 sunrpc af_packet nf_conntrack_netbios_ns ipt_REJECT nf_conntrack_ipv4 xt_state nf_conntrack nfnetlink iptable_filter ip_tables ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables ipv6 binfmt_misc thermal processor fan container nvram snd_intel8x0 snd_ac97_codec ac97_bus snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm evdev snd_timer snd soundcore snd_page_alloc intel_agp agpgart i2c_i801 ide_cd cdrom rtc unix
> CPU:    0
> EIP:    0060:[<c020cf15>]    Not tainted VLI
> 
> l *0xc020cf15
> 0xc020cf15 is in delay_tsc (/mnt/md0/devel/linux-mm/arch/i386/lib/delay.c:49).
> 44
> 45              rdtscl(bclock);
> 46              do {
> 47                      rep_nop();
> 48                      rdtscl(now);
> 49              } while ((now-bclock) < loops);
> 50      }
> 51
> 52      /*
> 53       * Since we calibrate only once at boot, this
> 
> 
> EFLAGS: 00200086   (2.6.21-rc5-mm4 #13)
> EIP is at delay_tsc+0xe/0x17
> eax: ec55c3c8   ebx: 00000001   ecx: ec55c348   edx: 00000086
> esi: 04531454   edi: 00000001   ebp: ce47fc48   esp: ce47fc44
> ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0068
> Process operapluginwrap (pid: 3560, ti=ce47e000 task=ccffd510 task.ti=ce47e000)
> Stack: c743bda0 ce47fc58 c020cedc 00000000 c021add9 ce47fc88 c021ade8 00000000 
>        00000002 00000001 c011df36 000f4240 000026eb a6ca16f0 c743bda0 c743bda0 
>        ce47fd18 ce47fca8 c034f4be 00000000 00000002 c011df36 c011fe36 c011f5b2 
> Call Trace:
>  [<c020cedc>] __delay+0xc/0xe
>  [<c021ade8>] _raw_spin_lock+0xc3/0x128
>  [<c034f4be>] _spin_lock+0x3c/0x42
>  [<c011df36>] task_rq_lock+0x36/0x70
>  [<c011f5bd>] try_to_wake_up+0x282/0x36b
>  [<c011f6b4>] default_wake_function+0xe/0x10
>  [<c0136fc2>] autoremove_wake_function+0x17/0x3a
>  [<c011c773>] __wake_up_common+0x32/0x55
>  [<c011d627>] __wake_up+0x31/0x42
>  [<c0125457>] wake_up_klogd+0x36/0x38
>  [<c020e38a>] bust_spinlocks+0x2a/0x2c
>  [<c01058bd>] die+0x16c/0x22c
>  [<c011b9ed>] do_page_fault+0x47c/0x55b
>  [<c034fdf1>] error_code+0x79/0x80
>  [<c011d840>] recalc_task_prio+0xfa/0x19e
>  [<c011da29>] enqueue_task+0x31/0x7d
>  [<c011f327>] activate_task+0xba/0xce
>  [<c01223f3>] wake_up_new_task+0x65/0x125
>  [<c012440d>] do_fork+0x10f/0x181
>  [<c010223f>] sys_clone+0x33/0x39
>  [<c010424c>] syscall_call+0x7/0xb
>  [<4cefd258>] 0x4cefd258
>  =======================
> INFO: lockdep is turned off.
> Code: 04 00 00 00 c1 e2 08 69 92 88 88 4e c0 fa 00 00 00 f7 e2 8d 42 01 e8 cb ff ff ff c9 c3 55 89 e5 53 89 c3 0f 31 89 c1 f3 90 0f 31 <29> c8 39 d8 72 f6 5b 5d c3 55 89 e5 83 ec 08 69 c0 1c 43 00 00 
> 
> I'll try without this patches
> 
> sched-fix-idle-load-balancing-in-softirqd-context.patch
> sched-fix-idle-load-balancing-in-softirqd-context-fix.patch
> sched-dynticks-idle-load-balancing-v3.patch
> speedup-divides-by-cpu_power-in-scheduler.patch
> sched-optimize-siblings-status-check-logic-in-wake_idle.patch
> #
> sched-dont-renice-kernel-threads.patch
> sched-remove-sleepavg-from-proc.patch
> sched-implement-staircase-deadline-cpu-scheduler.patch
> sched-implement-staircase-deadline-cpu-scheduler-misc-fixes.patch
> sched-implement-staircase-deadline-cpu-scheduler-staircase-improvements.patch
> sched-remove-noninteractive-flag.patch
> sched-document-sd-cpu-scheduler.patch
> #
> sched2-sched-domain-sysctl.patch
> #
> sched-add-above-background-load-function.patch
> 
> 
> http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.21-rc5-mm4/mm-oops
> http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.21-rc5-mm4/mm-config
> 

Sorry for a delay.

Con, I think that your
sched-implement-staircase-deadline-cpu-scheduler-staircase-improvements.patch
is causing this oops.

Regards,
Michal

-- 
Michal K. K. Piotrowski
LTG - Linux Testers Group (PL)
(http://www.stardust.webpages.pl/ltg/)
LTG - Linux Testers Group (EN)
(http://www.stardust.webpages.pl/linux_testers_group_en/)

Re: 2.6.21-rc5-mm4

[Con Kolivas]

On Wednesday 04 April 2007 08:20, Michal Piotrowski wrote:
> Michal Piotrowski napisał(a):
> > http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.21-rc5-mm4/m
> >m-oops
> > http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.21-rc5-mm4/m
> >m-config
>
> Sorry for a delay.

Never apologise! 

(I'm trying hard to stay offline for my own health so I may have huge delays).

> Con, I think that your
> sched-implement-staircase-deadline-cpu-scheduler-staircase-improvements.pat
>ch is causing this oops.

Thanks for heads up!

Try this patch please?
---
Fix exiting recalc_task_prio without p->array being updated.

Microoptimisation courtesy of Dmitry Adamushko <dmitry.adamushko@gmail.com>

Signed-off-by: Con Kolivas <kernel@kolivas.org>

---
 kernel/sched.c |   13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

Index: linux-2.6.21-rc5-mm3/kernel/sched.c
===================================================================
--- linux-2.6.21-rc5-mm3.orig/kernel/sched.c	2007-04-04 08:23:45.000000000 +1000
+++ linux-2.6.21-rc5-mm3/kernel/sched.c	2007-04-04 08:25:39.000000000 +1000
@@ -683,11 +683,13 @@ static void dequeue_task(struct task_str
  * The task is being queued on a fresh array so it has its entitlement
  * bitmap cleared.
  */
-static inline void task_new_array(struct task_struct *p, struct rq *rq)
+static void task_new_array(struct task_struct *p, struct rq *rq,
+			   struct prio_array *array)
 {
 	bitmap_zero(p->bitmap, PRIO_RANGE);
 	p->rotation = rq->prio_rotation;
 	p->time_slice = p->quota;
+	p->array = array;
 }
 
 /* Find the first slot from the relevant prio_matrix entry */
@@ -736,11 +738,8 @@ static inline int next_entitled_slot(str
 
 static void queue_expired(struct task_struct *p, struct rq *rq)
 {
-	p->array = rq->expired;
-	task_new_array(p, rq);
+	task_new_array(p, rq, rq->expired);
 	p->prio = p->normal_prio = first_prio_slot(p);
-	p->time_slice = p->quota;
-	p->rotation = rq->prio_rotation;
 }
 
 #ifdef CONFIG_SMP
@@ -800,9 +799,9 @@ static void recalc_task_prio(struct task
 			queue_expired(p, rq);
 			return;
 		} else
-			task_new_array(p, rq);
+			task_new_array(p, rq, array);
 	} else
-		task_new_array(p, rq);
+		task_new_array(p, rq, array);
 
 	queue_prio = next_entitled_slot(p, rq);
 	if (queue_prio >= MAX_PRIO) {

-- 
-ck

Re: 2.6.21-rc5-mm4

[Michal Piotrowski]

Con Kolivas napisał(a):
> On Wednesday 04 April 2007 08:20, Michal Piotrowski wrote:
>> Michal Piotrowski napisał(a):
>>> http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.21-rc5-mm4/m
>>> m-oops
>>> http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.21-rc5-mm4/m
>>> m-config
>> Sorry for a delay.
> 
> Never apologise! 
> 
> (I'm trying hard to stay offline for my own health so I may have huge delays).
> 
>> Con, I think that your
>> sched-implement-staircase-deadline-cpu-scheduler-staircase-improvements.pat
>> ch is causing this oops.
> 
> Thanks for heads up!
> 
> Try this patch please?

I still get the same oops.

BUG: unable to handle kernel paging request at virtual address 0743c798
 printing eip:
c011d80a
*pde = 00000000
Oops: 0000 [#1]
PREEMPT SMP 
last sysfs file: devices/platform/w83627hf.656/temp2_input
Modules linked in: ipt_MASQUERADE iptable_nat nf_nat nfsd exportfs lockd nfs_acl autofs4 sunrpc af_packet nf_conntrack_netbios_ns ipt_REJECT nf_conntrack_ipv4 xt_state nf_conntrack nfnetlink iptable_filter ip_tables ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables ipv6 binfmt_misc thermal processor fan container nvram snd_intel8x0 snd_ac97_codec ac97_bus snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd soundcore snd_page_alloc evdev i2c_i801 intel_agp agpgart ide_cd cdrom rtc unix
CPU:    0
EIP:    0060:[<c011d80a>]    Not tainted VLI
EFLAGS: 00210006   (2.6.21-rc5-mm4 #15)
EIP is at recalc_task_prio+0xc0/0x162

l *0xc011d80a
0xc011d80a is in recalc_task_prio (/mnt/md0/devel/linux-mm/kernel/sched.c:722).
717              * a queued better static_prio's prio_level.
718              */
719             if (p->static_prio < rq->best_static_prio) {
720                     search_prio = MAX_RT_PRIO;
721                     if (likely(p->policy != SCHED_BATCH))
722                             rq->best_static_prio = p->static_prio;
723             } else if (p->static_prio == rq->best_static_prio)
724                     search_prio = rq->prio_level[uprio];
725             else {
726                     search_prio = max(rq->prio_level[uprio],


eax: 1000008b   ebx: 10000027   ecx: 0000008b   edx: c743c27c
esi: cc33d510   edi: c743bda0   ebp: ce4a9ec0   esp: ce4a9e98
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0068
Process operapluginwrap (pid: 3559, ti=ce4a8000 task=ce006ac0 task.ti=ce4a8000)
Stack: 00000000 c743c27c 00000001 c011defe c01240b5 c04eac14 ce4a9ed0 cc33d510 
       c743bda0 cc33d510 ce4a9ed0 c011d9f1 00000001 cc33d510 ce4a9f00 c011f2ef 
       00000000 c052cda0 ce4a9f00 c743bda0 c8ada023 00000029 cc33d510 003d0f00 
Call Trace:
 [<c011d9f1>] enqueue_task+0x31/0x7d
 [<c011f2ef>] activate_task+0xba/0xce
 [<c01223bb>] wake_up_new_task+0x65/0x125
 [<c01243d5>] do_fork+0x10f/0x181
 [<c010223f>] sys_clone+0x33/0x39
 [<c010424c>] syscall_call+0x7/0xb
 [<4cefd258>] 0x4cefd258
 =======================
INFO: lockdep is turned off.
Code: 7d 29 31 d2 83 7e 5c 03 0f 95 c2 b8 74 4c 4a c0 e8 04 e4 0f 00 c7 45 e0 64 00 00 00 85 c0 74 23 8b 46 1c 89 87 fc 09 00 00 eb 18 <8b> 94 9f 5c 09 00 00 74 0c 8b 84 8f cc 07 00 00 39 d0 0f 4d d0 
EIP: [<c011d80a>] recalc_task_prio+0xc0/0x162 SS:ESP 0068:ce4a9e98
BUG: NMI Watchdog detected LOCKUP on CPU0, eip c021ad9a, registers:
Modules linked in: ipt_MASQUERADE iptable_nat nf_nat nfsd exportfs lockd nfs_acl autofs4 sunrpc af_packet nf_conntrack_netbios_ns ipt_REJECT nf_conntrack_ipv4 xt_state nf_conntrack nfnetlink iptable_filter ip_tables ip6t_REJECT xt_tcpudp ip6table_filter ip6_tables x_tables ipv6 binfmt_misc thermal processor fan container nvram snd_intel8x0 snd_ac97_codec ac97_bus snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd soundcore snd_page_alloc evdev i2c_i801 intel_agp agpgart ide_cd cdrom rtc unix
CPU:    0
EIP:    0060:[<c021ad9a>]    Not tainted VLI
EFLAGS: 00200006   (2.6.21-rc5-mm4 #15)
EIP is at _raw_spin_lock+0xb5/0x128

l *0xc021ad9a
0xc021ad9a is in _raw_spin_lock (/mnt/md0/devel/linux-mm/lib/spinlock_debug.c:109).
104             u64 loops = loops_per_jiffy * HZ;
105             int print_once = 1;
106
107             for (;;) {
108                     for (i = 0; i < loops; i++) {
109                             if (__raw_spin_trylock(&lock->raw_lock))
110                                     return;
111                             __delay(1);
112                     }
113                     /* lockup suspected: */


eax: 00000000   ebx: c743bda0   ecx: 88044b58   edx: 00000078
esi: 01f2bd6f   edi: 00000001   ebp: ce4a9c88   esp: ce4a9c60
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0068
Process operapluginwrap (pid: 3559, ti=ce4a8000 task=ce006ac0 task.ti=ce4a8000)
Stack: 00000000 00000002 00000001 c011defe 000f4240 000026ec a6ca8450 c743bda0 
       c743bda0 ce4a9d18 ce4a9ca8 c034f47e 00000000 00000002 c011defe c0140275 
       c0852720 c052cda0 ce4a9cc8 c011defe c0852720 000f4240 c9855510 c9799f00 
Call Trace:
 [<c034f47e>] _spin_lock+0x3c/0x42
 [<c011defe>] task_rq_lock+0x36/0x70
 [<c011f31d>] try_to_wake_up+0x1a/0x36b
 [<c011f67c>] default_wake_function+0xe/0x10
 [<c0136f8a>] autoremove_wake_function+0x17/0x3a
 [<c011c773>] __wake_up_common+0x32/0x55
 [<c011d62b>] __wake_up+0x31/0x42
 [<c012541f>] wake_up_klogd+0x36/0x38
 [<c020e34a>] bust_spinlocks+0x2a/0x2c
 [<c01058bd>] die+0x16c/0x22c
 [<c011b9ed>] do_page_fault+0x47c/0x55b
 [<c034fdb1>] error_code+0x79/0x80
 [<c011d80a>] recalc_task_prio+0xc0/0x162
 [<c011d9f1>] enqueue_task+0x31/0x7d
 [<c011f2ef>] activate_task+0xba/0xce
 [<c01223bb>] wake_up_new_task+0x65/0x125
 [<c01243d5>] do_fork+0x10f/0x181
 [<c010223f>] sys_clone+0x33/0x39
 [<c010424c>] syscall_call+0x7/0xb
 [<4cefd258>] 0x4cefd258
 =======================
INFO: lockdep is turned off.
Code: 9e c2 b8 38 16 4c c0 e8 7d 0e 00 00 85 c0 74 74 69 05 88 07 4a c0 e8 03 00 00 89 45 f0 bf 01 00 00 00 31 f6 eb 13 31 c0 86 03 46 <84> c0 7f 55 b8 01 00 00 00 e8 e8 20 ff ff 3b 75 f0 75 e8 85 ff 

serial console log
http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.21-rc5-mm4/mm-oops2

BTW. I want to do some scheduler benchmarks tomorrow.
Do sched-implement-staircase-deadline-cpu-scheduler-staircase-improvements.patch
is really important for staircase scheduler performance?

Regards,
Michal

-- 
Michal K. K. Piotrowski
LTG - Linux Testers Group (PL)
(http://www.stardust.webpages.pl/ltg/)
LTG - Linux Testers Group (EN)
(http://www.stardust.webpages.pl/linux_testers_group_en/)

[PATCH] sched: implement staircase deadline cpu scheduler improvements fix

[Con Kolivas]

On Wednesday 04 April 2007 09:31, Michal Piotrowski wrote:
> Con Kolivas napisał(a):
> > On Wednesday 04 April 2007 08:20, Michal Piotrowski wrote:
> >> Michal Piotrowski napisał(a):
> >>> http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.21-rc5-mm4
> >>>/m m-oops
> >>> http://www.stardust.webpages.pl/files/tbf/bitis-gabonica/2.6.21-rc5-mm4
> >>>/m m-config

> >> Con, I think that your
> >> sched-implement-staircase-deadline-cpu-scheduler-staircase-improvements.
> >>pat ch is causing this oops.
> >
> > Thanks for heads up!

Confirmed offline with Michal that the following patch fixes it. Thanks!

This should also make nice work better in the way the previous patch intended
it to.

---
Use of memset was bogus. Fix it.

Fix exiting recalc_task_prio without p->array being updated.

Microoptimisation courtesy of Dmitry Adamushko <dmitry.adamushko@gmail.com>

Signed-off-by: Con Kolivas <kernel@kolivas.org>

---
 kernel/sched.c |   17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)

Index: linux-2.6.21-rc5-mm4/kernel/sched.c
===================================================================
--- linux-2.6.21-rc5-mm4.orig/kernel/sched.c	2007-04-04 12:14:29.000000000 +1000
+++ linux-2.6.21-rc5-mm4/kernel/sched.c	2007-04-04 12:49:39.000000000 +1000
@@ -683,11 +683,13 @@ static void dequeue_task(struct task_str
  * The task is being queued on a fresh array so it has its entitlement
  * bitmap cleared.
  */
-static inline void task_new_array(struct task_struct *p, struct rq *rq)
+static void task_new_array(struct task_struct *p, struct rq *rq,
+			   struct prio_array *array)
 {
 	bitmap_zero(p->bitmap, PRIO_RANGE);
 	p->rotation = rq->prio_rotation;
 	p->time_slice = p->quota;
+	p->array = array;
 }
 
 /* Find the first slot from the relevant prio_matrix entry */
@@ -709,6 +711,8 @@ static inline int next_entitled_slot(str
 	DECLARE_BITMAP(tmp, PRIO_RANGE);
 	int search_prio, uprio = USER_PRIO(p->static_prio);
 
+	if (!rq->prio_level[uprio])
+		rq->prio_level[uprio] = MAX_RT_PRIO;
 	/*
 	 * Only priorities equal to the prio_level and above for their
 	 * static_prio are acceptable, and only if it's not better than
@@ -736,11 +740,8 @@ static inline int next_entitled_slot(str
 
 static void queue_expired(struct task_struct *p, struct rq *rq)
 {
-	p->array = rq->expired;
-	task_new_array(p, rq);
+	task_new_array(p, rq, rq->expired);
 	p->prio = p->normal_prio = first_prio_slot(p);
-	p->time_slice = p->quota;
-	p->rotation = rq->prio_rotation;
 }
 
 #ifdef CONFIG_SMP
@@ -800,9 +801,9 @@ static void recalc_task_prio(struct task
 			queue_expired(p, rq);
 			return;
 		} else
-			task_new_array(p, rq);
+			task_new_array(p, rq, array);
 	} else
-		task_new_array(p, rq);
+		task_new_array(p, rq, array);
 
 	queue_prio = next_entitled_slot(p, rq);
 	if (queue_prio >= MAX_PRIO) {
@@ -3445,7 +3446,7 @@ EXPORT_SYMBOL(sub_preempt_count);
 
 static inline void reset_prio_levels(struct rq *rq)
 {
-	memset(rq->prio_level, MAX_RT_PRIO, ARRAY_SIZE(rq->prio_level));
+	memset(rq->prio_level, 0, sizeof(int) * PRIO_RANGE);
 }
 
 /*

-- 
-ck

Re: 2.6.21-rc5-mm4

[Randy Dunlap]

On Mon, 2 Apr 2007 22:47:45 -0700 Andrew Morton wrote:

> 
> ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/
> 
> - The oops in git-net.patch has been fixed, so that tree has been restored. 
>   It is huge.
> 
> - Added the device-mapper development tree to the -mm lineup (Alasdair
>   Kergon).  It is a quilt tree, living at
>   ftp://ftp.kernel.org/pub/linux/kernel/people/agk/patches/2.6/editing/.
> 
> - Added davidel's signalfd stuff.
          ~~~~~~~~~~~~~~~~~~

with those new CONFIGs=n:  (we are discussing it)

arch/x86_64/kernel/built-in.o:(.rodata+0x18a8): undefined reference to `sys_signal
fd'
arch/x86_64/kernel/built-in.o:(.rodata+0x18b0): undefined reference to `sys_timerf
d'
arch/x86_64/kernel/built-in.o:(.rodata+0x18b8): undefined reference to `sys_eventf
d'
arch/x86_64/ia32/ia32entry.S:(.rodata+0xa08): undefined reference to `sys_signalfd
'
arch/x86_64/ia32/ia32entry.S:(.rodata+0xa10): undefined reference to `sys_timerfd'
arch/x86_64/ia32/ia32entry.S:(.rodata+0xa18): undefined reference to `sys_eventfd'
make: *** [.tmp_vmlinux1] Error 1

---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

Re: 2.6.21-rc5-mm4

[Davide Libenzi]

On Tue, 3 Apr 2007, Randy Dunlap wrote:

> On Mon, 2 Apr 2007 22:47:45 -0700 Andrew Morton wrote:
> 
> > 
> > ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/
> > 
> > - The oops in git-net.patch has been fixed, so that tree has been restored. 
> >   It is huge.
> > 
> > - Added the device-mapper development tree to the -mm lineup (Alasdair
> >   Kergon).  It is a quilt tree, living at
> >   ftp://ftp.kernel.org/pub/linux/kernel/people/agk/patches/2.6/editing/.
> > 
> > - Added davidel's signalfd stuff.
>           ~~~~~~~~~~~~~~~~~~
> 
> with those new CONFIGs=n:  (we are discussing it)
> 
> arch/x86_64/kernel/built-in.o:(.rodata+0x18a8): undefined reference to `sys_signal
> fd'
> arch/x86_64/kernel/built-in.o:(.rodata+0x18b0): undefined reference to `sys_timerf
> d'
> arch/x86_64/kernel/built-in.o:(.rodata+0x18b8): undefined reference to `sys_eventf
> d'
> arch/x86_64/ia32/ia32entry.S:(.rodata+0xa08): undefined reference to `sys_signalfd
> '
> arch/x86_64/ia32/ia32entry.S:(.rodata+0xa10): undefined reference to `sys_timerfd'
> arch/x86_64/ia32/ia32entry.S:(.rodata+0xa18): undefined reference to `sys_eventfd'
> make: *** [.tmp_vmlinux1] Error 1

I'll be sending Andrew patches against -mm4. The ones that fix the above, 
and the one for the include-files diet.



- Davide

Re: 2.6.21-rc5-mm4

[Badari Pulavarty]

On Mon, 2007-04-02 at 22:47 -0700, Andrew Morton wrote:
> ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/
> 

SLUB code hangs my ppc64 machine on boot.

Thanks,
Badari

booted from '/pci@800000020000003/pci@2,2/pci1069,b166@1/scsi@0/sd@4,0'
Enter "help" to get some basic usage information
boot: 2621rc5mm4
Please wait, loading kernel...
Allocated 0x00400000 bytes for executable @ 0x00400000
   Elf32 kernel loaded...

zImage starting: loaded at 0x00400000 (sp: 0x01a3fb10)
Allocating 0x822c40 bytes for kernel ...
OF version = 'IBM,SF225_096'
gunzipping (0x01c00000 <- 0x00408000:0x006a8d9b)...done 0x75cdf0 bytes
Finalizing device tree... using OF tree (promptr=00c39a50)
OF stdout device is: /vdevice/vty@30000000
Hypertas detected, assuming LPAR !
command line: root=/dev/sda2
memory layout at init:
  alloc_bottom : 0000000002427000
  alloc_top    : 0000000008000000
  alloc_top_hi : 00000001e8000000
  rmo_top      : 0000000008000000
  ram_top      : 00000001e8000000
Looking for displays
found display   : /pci@800000020000002/pci@2,2/pci@1/display@0,
opening ... done
instantiating rtas at 0x00000000077ca000 ... done
0000000000000000 : boot cpu     0000000000000000
0000000000000002 : starting cpu hw idx 0000000000000002... done
0000000000000004 : starting cpu hw idx 0000000000000004... done
0000000000000006 : starting cpu hw idx 0000000000000006... done
copying OF device tree ...
Building dt strings...
Building dt structure...
Device tree strings 0x0000000002428000 -> 0x00000000024292fe
Device tree struct  0x000000000242a000 -> 0x000000000243f000
Calling quiesce ...
returning from prom_init
Partition configured for 8 cpus.
Starting Linux PPC64 #1 SMP Tue Apr 3 10:42:07 PDT 2007
-----------------------------------------------------
ppc64_pft_size                = 0x1b
physicalMemorySize            = 0x1e8000000
ppc64_caches.dcache_line_size = 0x80
ppc64_caches.icache_line_size = 0x80
htab_address                  = 0x0000000000000000
htab_hash_mask                = 0xfffff
-----------------------------------------------------
Linux version 2.6.21-rc5-mm4-ppc64 (root@elm3a121) (gcc version 4.1.0
(SUSE Linux)) #1 SMP Tue Apr 3 10:42:07 PDT 2007
[boot]0012 Setup Arch
No ramdisk, default root is /dev/sda2
EEH: PCI Enhanced I/O Error Handling Enabled
PPC64 nvram contains 8192 bytes
Zone PFN ranges:
  DMA             0 ->  1998848
  Normal    1998848 ->  1998848
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0:        0 ->   974848
    1:   974848 ->  1998848
[boot]0015 Setup Done
Built 2 zonelists.  Total pages: 1971520
Kernel command line: root=/dev/sda2
[boot]0020 XICS Init
[boot]0021 XICS Done
PID hash table entries: 4096 (order: 12, 32768 bytes)
Console: colour dummy device 80x25
console handover: boot [udbg-1] -> real [hvc0]
Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
freeing bootmem node 0
freeing bootmem node 1
Memory: 7855400k/7995392k available (6060k kernel code, 139992k
reserved, 1236k data, 819k bss, 272k init)
SLUB V6: General Slabs=18, HW alignment=128, Processors=8, Nodes=16

Re: 2.6.21-rc5-mm4

[Christoph Lameter]

On Tue, 3 Apr 2007, Badari Pulavarty wrote:

> On Mon, 2007-04-02 at 22:47 -0700, Andrew Morton wrote:
> > ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/
> > 
> 
> SLUB code hangs my ppc64 machine on boot.

Could you enable NMI or something and get me a stackdump? PPC is notorious 
for having strange NUMA issues. Any fallbacks from node 0 on bootstrap? Is 
there memory on node 0?

Re: 2.6.21-rc5-mm4

[Badari Pulavarty]

On Tue, 2007-04-03 at 13:40 -0700, Christoph Lameter wrote:
> On Tue, 3 Apr 2007, Badari Pulavarty wrote:
> 
> > On Mon, 2007-04-02 at 22:47 -0700, Andrew Morton wrote:
> > > ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/
> > > 
> > 
> > SLUB code hangs my ppc64 machine on boot.
> 
> Could you enable NMI or something and get me a stackdump? PPC is notorious 
> for having strange NUMA issues. Any fallbacks from node 0 on bootstrap? Is 
> there memory on node 0?


Seems to be an issue with calibrate_delay() spinning in a tight
loop :(

BTW, machine boots fine with SLAB code - not sure why ?

Thanks,
Badari

SLUB V6: General Slabs=18, HW alignment=128, Processors=8, Nodes=16
cpu 0x0: Vector: 100 (System Reset) at [c000000000723bc0]
    pc: c00000000000999c: .calibrate_delay+0x90/0x1a8
    lr: c000000000009988: .calibrate_delay+0x7c/0x1a8
    sp: c000000000723e40
   msr: 8000000000081032
  current = 0xc00000000060ef50
  paca    = 0xc00000000060f800
    pid   = 0, comm = swapper
enter ? for help
[c000000000723ee0] c0000000005a7900 .start_kernel+0x334/0x3f4
[c000000000723f90] c000000000008524 .start_here_common+0x54/0x130
00 = 00000000ffff8adc   R16 = 0000000000000000
R01 = c000000000723e40   R17 = 0000000000000000
R02 = c0000000007207d8   R18 = 0000000000000000
R03 = 0000000000002000   R19 = 4000000001c00000
R04 = 0000000000000001   R20 = c0000000005dc618
R05 = 0000000000000001   R21 = 00000000021dc888
R06 = 0000000000000000   R22 = 0000000000000000
R07 = c00000000063fa38   R23 = c0000000005dc888
R08 = 000000000007ffff   R24 = 00000000021dc618
R09 = c000000000736400   R25 = 0000000001a3f9a0
R10 = c000000000766f80   R26 = c000000000758000
R11 = 00000000ffff8adc   R27 = c000000000759030
R12 = c000000000647ad0   R28 = c0000000007598b0
R13 = c00000000060f800   R29 = 0000000000000000
R14 = 0000000000000000   R30 = c0000000005f3db8
R15 = 0000000000000000   R31 = c000000000736400
pc  = c00000000000999c .calibrate_delay+0x90/0x1a8
lr  = c000000000009988 .calibrate_delay+0x7c/0x1a8
msr = 8000000000081032   cr  = 24002042
ctr = 0000000000000000   xer = 0000000000000005   trap =  100

Re: 2.6.21-rc5-mm4

[Christoph Lameter]

On Tue, 3 Apr 2007, Badari Pulavarty wrote:

> > > SLUB code hangs my ppc64 machine on boot.
> > 
> > Could you enable NMI or something and get me a stackdump? PPC is notorious 
> > for having strange NUMA issues. Any fallbacks from node 0 on bootstrap? Is 
> > there memory on node 0?
> 
> 
> Seems to be an issue with calibrate_delay() spinning in a tight
> loop :(
> 
> BTW, machine boots fine with SLAB code - not sure why ?

SLUB does not guarantee the special casing that SLAB provides for 
page sized allocations via kmalloc. SLAB aligns page sized allocations on 
page boundaries and does not allow full debugging of page sized 
allocations in order to keep these slabs on page sized boundaries.

I suspect this is related. I could add the same special casing to SLUB. 
But I really wish we would get rid of special slab behavior depending on 
the size of the slab. Maybe there is no way around it. Sigh.

What is the data that calibrate delay is looking at?

Re: 2.6.21-rc5-mm4

[Badari Pulavarty]

On Tue, 2007-04-03 at 15:59 -0700, Christoph Lameter wrote:
> On Tue, 3 Apr 2007, Badari Pulavarty wrote:
> 
> > > > SLUB code hangs my ppc64 machine on boot.
> > > 
> > > Could you enable NMI or something and get me a stackdump? PPC is notorious 
> > > for having strange NUMA issues. Any fallbacks from node 0 on bootstrap? Is 
> > > there memory on node 0?
> > 
> > 
> > Seems to be an issue with calibrate_delay() spinning in a tight
> > loop :(
> > 
> > BTW, machine boots fine with SLAB code - not sure why ?
> 
> SLUB does not guarantee the special casing that SLAB provides for 
> page sized allocations via kmalloc. SLAB aligns page sized allocations on 
> page boundaries and does not allow full debugging of page sized 
> allocations in order to keep these slabs on page sized boundaries.
> 
> I suspect this is related. I could add the same special casing to SLUB. 
> But I really wish we would get rid of special slab behavior depending on 
> the size of the slab. Maybe there is no way around it. Sigh.
> 
> What is the data that calibrate delay is looking at?


Its stuck in following .. 

                while ((loops_per_jiffy <<= 1) != 0) {
                        /* wait for "start of" clock tick */
                        ticks = jiffies;
                        while (ticks == jiffies)
                                /* nothing */;
                        /* Go .. */
                        ticks = jiffies;
                        __delay(loops_per_jiffy);
                        ticks = jiffies - ticks;
                        if (ticks)
                                break;
                }

Its stuck in (tight loop)

	while (ticks == jiffies)
                   /* nothing */;

I have no idea, what this code is supposed to be doing :(
Let me know, what I can do for you.

Thanks,
Badari

Re: 2.6.21-rc5-mm4

[Christoph Lameter]

On Tue, 3 Apr 2007, Badari Pulavarty wrote:

> I have no idea, what this code is supposed to be doing :(
> Let me know, what I can do for you.

I replicated the failure on IA64.... give me a minute ...

Re: 2.6.21-rc5-mm4

[Christoph Lameter]

On Tue, 3 Apr 2007, Badari Pulavarty wrote:

> Seems to be an issue with calibrate_delay() spinning in a tight
> loop :(
> 
> BTW, machine boots fine with SLAB code - not sure why ?

Interrupt disabled sigh.

Here is the fix:




SLUB: Fix numa bootstrap

NUMA bootstrap calls new_slab() if more than one node is found on bootup.
new_slab() assumes a standard slab context where interrupts must be
disabled. It enables interrupts for the call into the page allocator
and then disables them again. Interrupts do not have to be disabled
during on bootstrap because we still run single threaded there.

I dropped the interrupt preservation code just before SLUB v6 because
it looked useless there. SLUB worked on the following NUMA tests
that just had a single node. Sigh.

Enable interrupts after calling new_slab.

Signed-off-by: Christoph Lameter <clameter@sgi.com>

Index: linux-2.6.21-rc5-mm4/mm/slub.c
===================================================================
--- linux-2.6.21-rc5-mm4.orig/mm/slub.c	2007-04-03 18:07:41.000000000 -0700
+++ linux-2.6.21-rc5-mm4/mm/slub.c	2007-04-03 18:08:17.000000000 -0700
@@ -1436,6 +1436,8 @@ static int init_kmem_cache_nodes(struct 
 
 			BUG_ON(s->size < sizeof(struct kmem_cache_node));
 			page = new_slab(kmalloc_caches, gfpflags, node);
+			/* new_slab() disables interupts */
+			local_irq_enable();
 
 			BUG_ON(!page);
 			n = page->freelist;

Re: 2.6.21-rc5-mm4

[Badari Pulavarty]

On Tue, 2007-04-03 at 18:16 -0700, Christoph Lameter wrote:
> On Tue, 3 Apr 2007, Badari Pulavarty wrote:
> 
> > Seems to be an issue with calibrate_delay() spinning in a tight
> > loop :(
> > 
> > BTW, machine boots fine with SLAB code - not sure why ?
> 
> Interrupt disabled sigh.
> 
> Here is the fix:
> 
> 
> 
> 
> SLUB: Fix numa bootstrap
> 
> NUMA bootstrap calls new_slab() if more than one node is found on bootup.
> new_slab() assumes a standard slab context where interrupts must be
> disabled. It enables interrupts for the call into the page allocator
> and then disables them again. Interrupts do not have to be disabled
> during on bootstrap because we still run single threaded there.
> 
> I dropped the interrupt preservation code just before SLUB v6 because
> it looked useless there. SLUB worked on the following NUMA tests
> that just had a single node. Sigh.
> 
> Enable interrupts after calling new_slab.
> 
> Signed-off-by: Christoph Lameter <clameter@sgi.com>
> 
> Index: linux-2.6.21-rc5-mm4/mm/slub.c
> ===================================================================
> --- linux-2.6.21-rc5-mm4.orig/mm/slub.c	2007-04-03 18:07:41.000000000 -0700
> +++ linux-2.6.21-rc5-mm4/mm/slub.c	2007-04-03 18:08:17.000000000 -0700
> @@ -1436,6 +1436,8 @@ static int init_kmem_cache_nodes(struct 
>  
>  			BUG_ON(s->size < sizeof(struct kmem_cache_node));
>  			page = new_slab(kmalloc_caches, gfpflags, node);
> +			/* new_slab() disables interupts */
> +			local_irq_enable();
>  
>  			BUG_ON(!page);
>  			n = page->freelist;

Well !! Helps a little, but not enough to boot (hangs little later) :(
I will try to get stack trace for that.

Thanks,
Badari

boot: 2621rc5mm4
Please wait, loading kernel...
Allocated 0x00400000 bytes for executable @ 0x00400000
   Elf32 kernel loaded...

zImage starting: loaded at 0x00400000 (sp: 0x01a3fb10)
Allocating 0x826c40 bytes for kernel ...
OF version = 'IBM,SF225_096'
gunzipping (0x01c00000 <- 0x00408000:0x006a8e52)...done 0x760df0 bytes
Finalizing device tree... using OF tree (promptr=00c39a50)
OF stdout device is: /vdevice/vty@30000000
Hypertas detected, assuming LPAR !
command line: root=/dev/sda2
memory layout at init:
  alloc_bottom : 000000000242b000
  alloc_top    : 0000000008000000
  alloc_top_hi : 00000001e8000000
  rmo_top      : 0000000008000000
  ram_top      : 00000001e8000000
Looking for displays
found display   : /pci@800000020000002/pci@2,2/pci@1/display@0,
opening ... done
instantiating rtas at 0x00000000077ca000 ... done
0000000000000000 : boot cpu     0000000000000000
0000000000000002 : starting cpu hw idx 0000000000000002... done
0000000000000004 : starting cpu hw idx 0000000000000004... done
0000000000000006 : starting cpu hw idx 0000000000000006... done
copying OF device tree ...
Building dt strings...
Building dt structure...
Device tree strings 0x000000000242c000 -> 0x000000000242d2fe
Device tree struct  0x000000000242e000 -> 0x0000000002443000
Calling quiesce ...
returning from prom_init
Partition configured for 8 cpus.
Starting Linux PPC64 #7 SMP Wed Apr 4 07:52:49 PDT 2007
-----------------------------------------------------
ppc64_pft_size                = 0x1b
physicalMemorySize            = 0x1e8000000
ppc64_caches.dcache_line_size = 0x80
ppc64_caches.icache_line_size = 0x80
htab_address                  = 0x0000000000000000
htab_hash_mask                = 0xfffff
-----------------------------------------------------
Linux version 2.6.21-rc5-mm4-ppc64 (root@elm3a121) (gcc version 4.1.0
(SUSE Linux)) #7 SMP Wed Apr 4 07:52:49 PDT 2007
[boot]0012 Setup Arch
No ramdisk, default root is /dev/sda2
EEH: PCI Enhanced I/O Error Handling Enabled
PPC64 nvram contains 8192 bytes
Zone PFN ranges:
  DMA             0 ->  1998848
  Normal    1998848 ->  1998848
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0:        0 ->   974848
    1:   974848 ->  1998848
[boot]0015 Setup Done
Built 2 zonelists.  Total pages: 1971520
Kernel command line: root=/dev/sda2
[boot]0020 XICS Init
[boot]0021 XICS Done
PID hash table entries: 4096 (order: 12, 32768 bytes)
Console: colour dummy device 80x25
console handover: boot [udbg-1] -> real [hvc0]
Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
freeing bootmem node 0
freeing bootmem node 1
Memory: 7855384k/7995392k available (6064k kernel code, 140008k
reserved, 1236k data, 819k bss, 272k init)
SLUB V6: General Slabs=18, HW alignment=128, Processors=8, Nodes=16
Calibrating delay loop...475.13 BogoMIPS (lpj=2375680)
Security Framework v1.0.0 initialized
Mount-cache hash table entries: 256
Processor 1 found.
Processor 2 found.
Processor 3 found.
Processor 4 found.
Processor 5 found.
Processor 6 found.
Processor 7 found.
Brought up 8 CPUs
mm/memory.c:111: bad pud c0000000f20c0480.
could not vmalloc 20971520 bytes for cache!
------------[ cut here ]------------
Badness at mm/vmalloc.c:100
Call Trace:
[c0000000f20731f0] [c00000000001098c] .show_stack+0x68/0x1b0
(unreliable)
[c0000000f2073290] [c0000000001ed454] .report_bug+0x94/0xe8
[c0000000f2073320] [c00000000042a068] .program_check_exception
+0x178/0x634
[c0000000f20733d0] [c0000000000046f4] program_check_common+0xf4/0x100
--- Exception: 700 at .map_vm_area+0x1b0/0x324
    LR = .__vmalloc_area_node+0x198/0x1ec
[c0000000f20736c0] [ffffffffffffffff] 0xffffffffffffffff (unreliable)
[c0000000f20737a0] [c0000000000c7538] .__vmalloc_area_node+0x198/0x1ec
[c0000000f2073870] [c0000000000c740c] .__vmalloc_area_node+0x6c/0x1ec
[c0000000f2073940] [c000000000059580] .arch_init_sched_domains
+0xb9c/0x10b0
[c0000000f2073d80] [c0000000005c330c] .sched_init_smp+0x60/0x430
[c0000000f2073ea0] [c0000000005a8b18] .kernel_init+0x158/0x3c0
[c0000000f2073f90] [c00000000002899c] .kernel_thread+0x4c/0x68
------------[ cut here ]------------
Badness at mm/vmalloc.c:100
Call Trace:
[c0000000f20731f0] [c00000000001098c] .show_stack+0x68/0x1b0
(unreliable)
[c0000000f2073290] [c0000000001ed454] .report_bug+0x94/0xe8
[c0000000f2073320] [c00000000042a068] .program_check_exception
+0x178/0x634
[c0000000f20733d0] [c0000000000046f4] program_check_common+0xf4/0x100
--- Exception: 700 at .map_vm_area+0x1b0/0x324
    LR = .__vmalloc_area_node+0x198/0x1ec
[c0000000f20736c0] [ffffffffffffffff] 0xffffffffffffffff (unreliable)
[c0000000f20737a0] [c0000000000c7538] .__vmalloc_area_node+0x198/0x1ec
[c0000000f2073870] [c0000000000c740c] .__vmalloc_area_node+0x6c/0x1ec
[c0000000f2073940] [c000000000059580] .arch_init_sched_domains
+0xb9c/0x10b0
[c0000000f2073d80] [c0000000005c330c] .sched_init_smp+0x60/0x430
[c0000000f2073ea0] [c0000000005a8b18] .kernel_init+0x158/0x3c0
[c0000000f2073f90] [c00000000002899c] .kernel_thread+0x4c/0x68
------------[ cut here ]------------
Badness at mm/vmalloc.c:100
Call Trace:
[c0000000f20731f0] [c00000000001098c] .show_stack+0x68/0x1b0
(unreliable)
[c0000000f2073290] [c0000000001ed454] .report_bug+0x94/0xe8
[c0000000f2073320] [c00000000042a068] .program_check_exception
+0x178/0x634
[c0000000f20733d0] [c0000000000046f4] program_check_common+0xf4/0x100
--- Exception: 700 at .map_vm_area+0x1b0/0x324
    LR = .__vmalloc_area_node+0x198/0x1ec
[c0000000f20736c0] [ffffffffffffffff] 0xffffffffffffffff (unreliable)
[c0000000f20737a0] [c0000000000c7538] .__vmalloc_area_node+0x198/0x1ec
[c0000000f2073870] [c0000000000c740c] .__vmalloc_area_node+0x6c/0x1ec
[c0000000f2073940] [c000000000059580] .arch_init_sched_domains
+0xb9c/0x10b0
[c0000000f2073d80] [c0000000005c330c] .sched_init_smp+0x60/0x430
[c0000000f2073ea0] [c0000000005a8b18] .kernel_init+0x158/0x3c0
[c0000000f2073f90] [c00000000002899c] .kernel_thread+0x4c/0x68
------------[ cut here ]------------
Badness at mm/vmalloc.c:100
Call Trace:
[c0000000f20731f0] [c00000000001098c] .show_stack+0x68/0x1b0
(unreliable)
[c0000000f2073290] [c0000000001ed454] .report_bug+0x94/0xe8
[c0000000f2073320] [c00000000042a068] .program_check_exception
+0x178/0x634
[c0000000f20733d0] [c0000000000046f4] program_check_common+0xf4/0x100
--- Exception: 700 at .map_vm_area+0x1b0/0x324
    LR = .__vmalloc_area_node+0x198/0x1ec
[c0000000f20736c0] [ffffffffffffffff] 0xffffffffffffffff (unreliable)
[c0000000f20737a0] [c0000000000c7538] .__vmalloc_area_node+0x198/0x1ec
[c0000000f2073870] [c0000000000c740c] .__vmalloc_area_node+0x6c/0x1ec
[c0000000f2073940] [c000000000059580] .arch_init_sched_domains
+0xb9c/0x10b0
[c0000000f2073d80] [c0000000005c330c] .sched_init_smp+0x60/0x430
[c0000000f2073ea0] [c0000000005a8b18] .kernel_init+0x158/0x3c0
[c0000000f2073f90] [c00000000002899c] .kernel_thread+0x4c/0x68
------------[ cut here ]------------
Badness at mm/vmalloc.c:100
Call Trace:
[c0000000f20731f0] [c00000000001098c] .show_stack+0x68/0x1b0
(unreliable)
[c0000000f2073290] [c0000000001ed454] .report_bug+0x94/0xe8
[c0000000f2073320] [c00000000042a068] .program_check_exception
+0x178/0x634
[c0000000f20733d0] [c0000000000046f4] program_check_common+0xf4/0x100
--- Exception: 700 at .map_vm_area+0x1b0/0x324
    LR = .__vmalloc_area_node+0x198/0x1ec
[c0000000f20736c0] [ffffffffffffffff] 0xffffffffffffffff (unreliable)
[c0000000f20737a0] [c0000000000c7538] .__vmalloc_area_node+0x198/0x1ec
[c0000000f2073870] [c0000000000c740c] .__vmalloc_area_node+0x6c/0x1ec
[c0000000f2073940] [c000000000059580] .arch_init_sched_domains
+0xb9c/0x10b0
[c0000000f2073d80] [c0000000005c330c] .sched_init_smp+0x60/0x430
[c0000000f2073ea0] [c0000000005a8b18] .kernel_init+0x158/0x3c0
[c0000000f2073f90] [c00000000002899c] .kernel_thread+0x4c/0x68
could not vmalloc 20971520 bytes for cache!
migration_cost=0,1000,1000
NET: Registered protocol family 16
IOMMU table initialized, virtual merging enabled
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
NET: Registered protocol family 2
IP route cache hash table entries: 262144 (order: 9, 2097152 bytes)
TCP established hash table entries: 524288 (order: 11, 12582912 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 524288 bind 65536)
TCP reno registered
IBM eBus Device Driver
audit: initializing netlink socket (disabled)
audit(1175698620.610:1): initialized
Total HugeTLB memory allocated, 0
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
mm/memory.c:111: bad pud c0000000f20c1200.
mm/memory.c:111: bad pud c0000000f20c1680.
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
rpaphp: RPA HOT Plug PCI Controller Driver version: 0.1
rpaphp: Slot [0001:00:02.0](PCI location=U7879.001.DQD02PW-P1-C3)
registered
rpaphp: Slot [0001:00:02.2](PCI location=U7879.001.DQD02PW-P1-C4)
registered
rpaphp: Slot [0001:00:02.4](PCI location=U7879.001.DQD02PW-P1-C5)
registered
rpaphp: Slot [0001:00:02.6](PCI location=U7879.001.DQD02PW-P1-C6)
registered
rpaphp: Slot [0002:00:02.0](PCI location=U7879.001.DQD02PW-P1-C1)
registered
rpaphp: Slot [0002:00:02.6](PCI location=U7879.001.DQD02PW-P1-C2)
registered
matroxfb: Matrox G450 detected
PInS data found at offset 31168
PInS memtype = 5
matroxfb: 640x480x8bpp (virtual: 640x26214)
matroxfb: framebuffer at 0x40170000000, mapped to 0xd000080080004000,
size 33554432

Re: 2.6.21-rc5-mm4

[Badari Pulavarty]

On Wed, 2007-04-04 at 08:12 -0700, Badari Pulavarty wrote:
> On Tue, 2007-04-03 at 18:16 -0700, Christoph Lameter wrote:
> > On Tue, 3 Apr 2007, Badari Pulavarty wrote:
> > 
> > > Seems to be an issue with calibrate_delay() spinning in a tight
> > > loop :(
> > > 
> > > BTW, machine boots fine with SLAB code - not sure why ?
> > 
> > Interrupt disabled sigh.
> > 
> > Here is the fix:
> > 
> > 
> > 
> > 
> > SLUB: Fix numa bootstrap
> > 
> > NUMA bootstrap calls new_slab() if more than one node is found on bootup.
> > new_slab() assumes a standard slab context where interrupts must be
> > disabled. It enables interrupts for the call into the page allocator
> > and then disables them again. Interrupts do not have to be disabled
> > during on bootstrap because we still run single threaded there.
> > 
> > I dropped the interrupt preservation code just before SLUB v6 because
> > it looked useless there. SLUB worked on the following NUMA tests
> > that just had a single node. Sigh.
> > 
> > Enable interrupts after calling new_slab.
> > 
> > Signed-off-by: Christoph Lameter <clameter@sgi.com>
> > 
> > Index: linux-2.6.21-rc5-mm4/mm/slub.c
> > ===================================================================
> > --- linux-2.6.21-rc5-mm4.orig/mm/slub.c	2007-04-03 18:07:41.000000000 -0700
> > +++ linux-2.6.21-rc5-mm4/mm/slub.c	2007-04-03 18:08:17.000000000 -0700
> > @@ -1436,6 +1436,8 @@ static int init_kmem_cache_nodes(struct 
> >  
> >  			BUG_ON(s->size < sizeof(struct kmem_cache_node));
> >  			page = new_slab(kmalloc_caches, gfpflags, node);
> > +			/* new_slab() disables interupts */
> > +			local_irq_enable();
> >  
> >  			BUG_ON(!page);
> >  			n = page->freelist;
> 
> Well !! Helps a little, but not enough to boot (hangs little later) :(
> I will try to get stack trace for that.

Better debug with slub_debug.
Hope this helps.

Thanks,
Badari

boot: 2621rc5mm4 xmon=on slub_debug
Please wait, loading kernel...
Allocated 0x00400000 bytes for executable @ 0x00400000
   Elf32 kernel loaded...

zImage starting: loaded at 0x00400000 (sp: 0x01a3fb10)
Allocating 0x826c40 bytes for kernel ...
OF version = 'IBM,SF225_096'
gunzipping (0x01c00000 <- 0x00408000:0x006a8e52)...done 0x760df0 bytes
Finalizing device tree... using OF tree (promptr=00c39a50)
OF stdout device is: /vdevice/vty@30000000
Hypertas detected, assuming LPAR !
command line: root=/dev/sda2 xmon=on slub_debug
memory layout at init:
  alloc_bottom : 000000000242b000
  alloc_top    : 0000000008000000
  alloc_top_hi : 00000001e8000000
  rmo_top      : 0000000008000000
  ram_top      : 00000001e8000000
Looking for displays
found display   : /pci@800000020000002/pci@2,2/pci@1/display@0,
opening ... done
instantiating rtas at 0x00000000077ca000 ... done
0000000000000000 : boot cpu     0000000000000000
0000000000000002 : starting cpu hw idx 0000000000000002... done
0000000000000004 : starting cpu hw idx 0000000000000004... done
0000000000000006 : starting cpu hw idx 0000000000000006... done
copying OF device tree ...
Building dt strings...
Building dt structure...
Device tree strings 0x000000000242c000 -> 0x000000000242d2fe
Device tree struct  0x000000000242e000 -> 0x0000000002443000
Calling quiesce ...
returning from prom_init
Partition configured for 8 cpus.
Starting Linux PPC64 #7 SMP Wed Apr 4 07:52:49 PDT 2007
-----------------------------------------------------
ppc64_pft_size                = 0x1b
physicalMemorySize            = 0x1e8000000
ppc64_caches.dcache_line_size = 0x80
ppc64_caches.icache_line_size = 0x80
htab_address                  = 0x0000000000000000
htab_hash_mask                = 0xfffff
-----------------------------------------------------
Linux version 2.6.21-rc5-mm4-ppc64 (root@elm3a121) (gcc version 4.1.0
(SUSE Linux)) #7 SMP Wed Apr 4 07:52:49 PDT 2007
[boot]0012 Setup Arch
No ramdisk, default root is /dev/sda2
EEH: PCI Enhanced I/O Error Handling Enabled
PPC64 nvram contains 8192 bytes
Zone PFN ranges:
  DMA             0 ->  1998848
  Normal    1998848 ->  1998848
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0:        0 ->   974848
    1:   974848 ->  1998848
[boot]0015 Setup Done
Built 2 zonelists.  Total pages: 1971520
Kernel command line: root=/dev/sda2 xmon=on slub_debug
[boot]0020 XICS Init
[boot]0021 XICS Done
PID hash table entries: 4096 (order: 12, 32768 bytes)
Console: colour dummy device 80x25
console handover: boot [udbg-1] -> real [hvc0]
Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
freeing bootmem node 0
freeing bootmem node 1
Memory: 7855384k/7995392k available (6064k kernel code, 140008k
reserved, 1236k data, 819k bss, 272k init)
SLUB V6: General Slabs=18, HW alignment=128, Processors=8, Nodes=16
Calibrating delay loop... 475.13 BogoMIPS (lpj=2375680)
Security Framework v1.0.0 initialized
Mount-cache hash table entries: 256
Processor 1 found.
Processor 2 found.
Processor 3 found.
Processor 4 found.
Processor 5 found.
Processor 6 found.
Processor 7 found.
Brought up 8 CPUs
mm/memory.c:111: bad pud c000000005100480.
could not vmalloc 20971520 bytes for cache!
*** SLUB: Redzone Inactive check fails in kmalloc-64@0xc000000005081180
Slab 0xc000000000941eb8
    offset=384 flags=0x5000000000c7 inuse=3 freelist=0xc000000005081180
  Bytes b4 0xc000000005081170:  00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 ................
    Object 0xc000000005081180:  00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 ................
    Object 0xc000000005081190:  00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 ................
    Object 0xc0000000050811a0:  00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 ................
    Object 0xc0000000050811b0:  00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 ................
   Redzone 0xc0000000050811c0:  00 00 00 00 00 00 00
00                         ........
FreePointer 0xc0000000050811c8 -> 0x0000000000000000
Call Trace:
[c0000000f20974c0] [c00000000001098c] .show_stack+0x68/0x1b0
(unreliable)
[c0000000f2097560] [c0000000000d8ef0] .object_err+0x1bc/0x1e4
[c0000000f2097600] [c0000000000d91c8] .check_object+0x140/0x220
[c0000000f20976a0] [c0000000000d9f98] .alloc_object_checks+0xc0/0x150
[c0000000f2097730] [c0000000000da318] .kmem_cache_alloc_node+0x2f0/0x34c
[c0000000f20977f0] [c0000000000c69f4] .__get_vm_area_node+0xb0/0x208
[c0000000f20978b0] [c0000000000c75e8] .__vmalloc_node+0x5c/0xb4
[c0000000f2097940] [c000000000059580] .arch_init_sched_domains
+0xb9c/0x10b0
[c0000000f2097d80] [c0000000005c330c] .sched_init_smp+0x60/0x430
[c0000000f2097ea0] [c0000000005a8b18] .kernel_init+0x158/0x3c0
[c0000000f2097f90] [c00000000002899c] .kernel_thread+0x4c/0x68
------------[ cut here ]------------
Badness at mm/vmalloc.c:100
Call Trace:
[c0000000f20971f0] [c00000000001098c] .show_stack+0x68/0x1b0
(unreliable)
[c0000000f2097290] [c0000000001ed454] .report_bug+0x94/0xe8
[c0000000f2097320] [c00000000042a068] .program_check_exception
+0x178/0x634
[c0000000f20973d0] [c0000000000046f4] program_check_common+0xf4/0x100
--- Exception: 700 at .map_vm_area+0x1b0/0x324
    LR = .__vmalloc_area_node+0x198/0x1ec
[c0000000f20976c0] [ffffffffffffffff] 0xffffffffffffffff (unreliable)
[c0000000f20977a0] [c0000000000c7538] .__vmalloc_area_node+0x198/0x1ec
[c0000000f2097870] [c0000000000c740c] .__vmalloc_area_node+0x6c/0x1ec
[c0000000f2097940] [c000000000059580] .arch_init_sched_domains
+0xb9c/0x10b0
[c0000000f2097d80] [c0000000005c330c] .sched_init_smp+0x60/0x430
[c0000000f2097ea0] [c0000000005a8b18] .kernel_init+0x158/0x3c0
[c0000000f2097f90] [c00000000002899c] .kernel_thread+0x4c/0x68
------------[ cut here ]------------
Badness at mm/vmalloc.c:100
Call Trace:
[c0000000f20971f0] [c00000000001098c] .show_stack+0x68/0x1b0
(unreliable)
[c0000000f2097290] [c0000000001ed454] .report_bug+0x94/0xe8
[c0000000f2097320] [c00000000042a068] .program_check_exception
+0x178/0x634
[c0000000f20973d0] [c0000000000046f4] program_check_common+0xf4/0x100
--- Exception: 700 at .map_vm_area+0x1b0/0x324
    LR = .__vmalloc_area_node+0x198/0x1ec
[c0000000f20976c0] [ffffffffffffffff] 0xffffffffffffffff (unreliable)
[c0000000f20977a0] [c0000000000c7538] .__vmalloc_area_node+0x198/0x1ec
[c0000000f2097870] [c0000000000c740c] .__vmalloc_area_node+0x6c/0x1ec
[c0000000f2097940] [c000000000059580] .arch_init_sched_domains
+0xb9c/0x10b0
[c0000000f2097d80] [c0000000005c330c] .sched_init_smp+0x60/0x430
[c0000000f2097ea0] [c0000000005a8b18] .kernel_init+0x158/0x3c0
[c0000000f2097f90] [c00000000002899c] .kernel_thread+0x4c/0x68
------------[ cut here ]------------
Badness at mm/vmalloc.c:100
Call Trace:
[c0000000f20971f0] [c00000000001098c] .show_stack+0x68/0x1b0
(unreliable)
[c0000000f2097290] [c0000000001ed454] .report_bug+0x94/0xe8
[c0000000f2097320] [c00000000042a068] .program_check_exception
+0x178/0x634
[c0000000f20973d0] [c0000000000046f4] program_check_common+0xf4/0x100
--- Exception: 700 at .map_vm_area+0x1b0/0x324
    LR = .__vmalloc_area_node+0x198/0x1ec
[c0000000f20976c0] [ffffffffffffffff] 0xffffffffffffffff (unreliable)
[c0000000f20977a0] [c0000000000c7538] .__vmalloc_area_node+0x198/0x1ec
[c0000000f2097870] [c0000000000c740c] .__vmalloc_area_node+0x6c/0x1ec
[c0000000f2097940] [c000000000059580] .arch_init_sched_domains
+0xb9c/0x10b0
[c0000000f2097d80] [c0000000005c330c] .sched_init_smp+0x60/0x430
[c0000000f2097ea0] [c0000000005a8b18] .kernel_init+0x158/0x3c0
[c0000000f2097f90] [c00000000002899c] .kernel_thread+0x4c/0x68
------------[ cut here ]------------
Badness at mm/vmalloc.c:100
Call Trace:
[c0000000f20971f0] [c00000000001098c] .show_stack+0x68/0x1b0
(unreliable)
[c0000000f2097290] [c0000000001ed454] .report_bug+0x94/0xe8
[c0000000f2097320] [c00000000042a068] .program_check_exception
+0x178/0x634
[c0000000f20973d0] [c0000000000046f4] program_check_common+0xf4/0x100
--- Exception: 700 at .map_vm_area+0x1b0/0x324
    LR = .__vmalloc_area_node+0x198/0x1ec
[c0000000f20976c0] [ffffffffffffffff] 0xffffffffffffffff (unreliable)
[c0000000f20977a0] [c0000000000c7538] .__vmalloc_area_node+0x198/0x1ec
[c0000000f2097870] [c0000000000c740c] .__vmalloc_area_node+0x6c/0x1ec
[c0000000f2097940] [c000000000059580] .arch_init_sched_domains
+0xb9c/0x10b0
[c0000000f2097d80] [c0000000005c330c] .sched_init_smp+0x60/0x430
[c0000000f2097ea0] [c0000000005a8b18] .kernel_init+0x158/0x3c0
[c0000000f2097f90] [c00000000002899c] .kernel_thread+0x4c/0x68
------------[ cut here ]------------
Badness at mm/vmalloc.c:100
Call Trace:
[c0000000f20971f0] [c00000000001098c] .show_stack+0x68/0x1b0
(unreliable)
[c0000000f2097290] [c0000000001ed454] .report_bug+0x94/0xe8
[c0000000f2097320] [c00000000042a068] .program_check_exception
+0x178/0x634
[c0000000f20973d0] [c0000000000046f4] program_check_common+0xf4/0x100
--- Exception: 700 at .map_vm_area+0x1b0/0x324
    LR = .__vmalloc_area_node+0x198/0x1ec
[c0000000f20976c0] [ffffffffffffffff] 0xffffffffffffffff (unreliable)
[c0000000f20977a0] [c0000000000c7538] .__vmalloc_area_node+0x198/0x1ec
[c0000000f2097870] [c0000000000c740c] .__vmalloc_area_node+0x6c/0x1ec
[c0000000f2097940] [c000000000059580] .arch_init_sched_domains
+0xb9c/0x10b0
[c0000000f2097d80] [c0000000005c330c] .sched_init_smp+0x60/0x430
[c0000000f2097ea0] [c0000000005a8b18] .kernel_init+0x158/0x3c0
[c0000000f2097f90] [c00000000002899c] .kernel_thread+0x4c/0x68
Unable to handle kernel paging request for data at address
0x500000000008
Faulting instruction address: 0xc0000000000c7164
cpu 0x4: Vector: 300 (Data Access) at [c0000000f20975a0]
    pc: c0000000000c7164: .remove_vm_area+0x38/0xb8
    lr: c0000000000c7158: .remove_vm_area+0x2c/0xb8
    sp: c0000000f2097820
   msr: 8000000000009032
   dar: 500000000008
 dsisr: 40000000
  current = 0xc0000000050a0000
  paca    = 0xc000000000611000
    pid   = 1, comm = swapper
------------[ cut here ]------------
Badness at arch/powerpc/kernel/entry_64.S:651
Call Trace:
[c0000000f20969f0] [c00000000001098c] .show_stack+0x68/0x1b0
(unreliable)
[c0000000f2096a90] [c0000000001ed454] .report_bug+0x94/0xe8
[c0000000f2096b20] [c00000000042a068] .program_check_exception
+0x178/0x634
[c0000000f2096bd0] [c0000000000046f4] program_check_common+0xf4/0x100
--- Exception: 700 at .enter_rtas+0xa0/0x10c
    LR = .xmon_core+0x584/0x934
[c0000000f2096ec0] [0000000000000000] 0x0 (unreliable)
[c0000000f20970a0] [c000000000051ac0] .xmon_core+0x584/0x934
[c0000000f2097230] [c0000000000520c0] .xmon+0x38/0x4c
[c0000000f2097410] [c000000000026dc8] .die+0x58/0x264
[c0000000f20974b0] [c00000000002fce4] .bad_page_fault+0xb8/0xd4
[c0000000f2097530] [c000000000004b18] handle_page_fault+0x3c/0x58
--- Exception: 300 at .remove_vm_area+0x38/0xb8
    LR = .remove_vm_area+0x2c/0xb8
[c0000000f20978b0] [c0000000000c7234] .__vunmap+0x50/0x100
[c0000000f2097940] [c0000000000597cc] .arch_init_sched_domains
+0xde8/0x10b0
[c0000000f2097d80] [c0000000005c330c] .sched_init_smp+0x60/0x430
[c0000000f2097ea0] [c0000000005a8b18] .kernel_init+0x158/0x3c0
[c0000000f2097f90] [c00000000002899c] .kernel_thread+0x4c/0x68
enter ? for help
[c0000000f20978b0] c0000000000c7234 .__vunmap+0x50/0x100
[c0000000f2097940] c0000000000597cc .arch_init_sched_domains
+0xde8/0x10b0
[c0000000f2097d80] c0000000005c330c .sched_init_smp+0x60/0x430
[c0000000f2097ea0] c0000000005a8b18 .kernel_init+0x158/0x3c0
[c0000000f2097f90] c00000000002899c .kernel_thread+0x4c/0x68
4:mon>

Re: 2.6.21-rc5-mm4 (SLUB powerpc)

[Christoph Lameter]

On Wed, 4 Apr 2007, Badari Pulavarty wrote:

> Well !! Helps a little, but not enough to boot (hangs little later) :(
> I will try to get stack trace for that.

Great! Thanks for all the debugging help.

 
> Processor 6 found.
> Processor 7 found.
> Brought up 8 CPUs
> mm/memory.c:111: bad pud c0000000f20c0480.

Hmmm... Checking for slabs used in powerpc arch code:

The pgtable cache is configured as


          pgtable_cache[i] = kmem_cache_create(name,
                                                     size, size,
                                                     SLAB_HWCACHE_ALIGN |
                                                     SLAB_MUST_HWCACHE_ALIGN,
                                                     zero_ctor,
                                                     NULL);

Hmmm.... aligned slabs at size and then we MUST_HWCACHE_ALIGN?? Two 
competing alignment requirements and a constructor. Constructor requires
the moving of the free pointer after the slab and thus increases the slab 
size.

Sigh. IF SLAB_HWCACHE_ALIGN is set then SLUB believes this to be the 
ultimate demand that overrides all other alignments and only aligns to the 
cacheline. Try the following fix:



SLUB: Treat SLAB_HWCACHE_ALIGN as a mininum and not as *the* alignment

If the specified alignment is higher than L1_CACHE_BYTES and
SLAB_HWCACHE_ALIGN is set then use the higher alignment.

Signed-off-by: Christoph Lameter <clameter@sgi.com>

Index: linux-2.6.21-rc5-mm4/mm/slub.c
===================================================================
--- linux-2.6.21-rc5-mm4.orig/mm/slub.c	2007-04-04 10:09:20.000000000 -0700
+++ linux-2.6.21-rc5-mm4/mm/slub.c	2007-04-04 10:09:42.000000000 -0700
@@ -1373,10 +1373,7 @@ static int calculate_order(int size)
 static unsigned long calculate_alignment(unsigned long flags,
 		unsigned long align)
 {
-	if (flags & SLAB_HWCACHE_ALIGN)
-		return L1_CACHE_BYTES;
-
-	if (flags & SLAB_MUST_HWCACHE_ALIGN)
+	if (flags & (SLAB_MUST_HWCACHE_ALIGN | SLAB_HWCACHE_ALIGN))
 		return max_t(unsigned long, align, L1_CACHE_BYTES);
 
 	if (align < ARCH_SLAB_MINALIGN)

Re: 2.6.21-rc5-mm4 (SLUB powerpc)

[Badari Pulavarty]

On Wed, 2007-04-04 at 10:13 -0700, Christoph Lameter wrote:
> On Wed, 4 Apr 2007, Badari Pulavarty wrote:
> 
> > Well !! Helps a little, but not enough to boot (hangs little later) :(
> > I will try to get stack trace for that.
> 
> Great! Thanks for all the debugging help.
> 
>  
> > Processor 6 found.
> > Processor 7 found.
> > Brought up 8 CPUs
> > mm/memory.c:111: bad pud c0000000f20c0480.
> 
> Hmmm... Checking for slabs used in powerpc arch code:
> 
> The pgtable cache is configured as
> 
> 
>           pgtable_cache[i] = kmem_cache_create(name,
>                                                      size, size,
>                                                      SLAB_HWCACHE_ALIGN |
>                                                      SLAB_MUST_HWCACHE_ALIGN,
>                                                      zero_ctor,
>                                                      NULL);
> 
> Hmmm.... aligned slabs at size and then we MUST_HWCACHE_ALIGN?? Two 
> competing alignment requirements and a constructor. Constructor requires
> the moving of the free pointer after the slab and thus increases the slab 
> size.
> 
> Sigh. IF SLAB_HWCACHE_ALIGN is set then SLUB believes this to be the 
> ultimate demand that overrides all other alignments and only aligns to the 
> cacheline. Try the following fix:
> 
> 
> 
> SLUB: Treat SLAB_HWCACHE_ALIGN as a mininum and not as *the* alignment
> 
> If the specified alignment is higher than L1_CACHE_BYTES and
> SLAB_HWCACHE_ALIGN is set then use the higher alignment.
> 
> Signed-off-by: Christoph Lameter <clameter@sgi.com>
> 
> Index: linux-2.6.21-rc5-mm4/mm/slub.c
> ===================================================================
> --- linux-2.6.21-rc5-mm4.orig/mm/slub.c	2007-04-04 10:09:20.000000000 -0700
> +++ linux-2.6.21-rc5-mm4/mm/slub.c	2007-04-04 10:09:42.000000000 -0700
> @@ -1373,10 +1373,7 @@ static int calculate_order(int size)
>  static unsigned long calculate_alignment(unsigned long flags,
>  		unsigned long align)
>  {
> -	if (flags & SLAB_HWCACHE_ALIGN)
> -		return L1_CACHE_BYTES;
> -
> -	if (flags & SLAB_MUST_HWCACHE_ALIGN)
> +	if (flags & (SLAB_MUST_HWCACHE_ALIGN | SLAB_HWCACHE_ALIGN))
>  		return max_t(unsigned long, align, L1_CACHE_BYTES);
>  
>  	if (align < ARCH_SLAB_MINALIGN)

Next issue ? Sorry.

Thanks,
Badari

Allocated 0x00400000 bytes for executable @ 0x00400000
   Elf32 kernel loaded...

zImage starting: loaded at 0x00400000 (sp: 0x01a3fb10)
Allocating 0x822c40 bytes for kernel ...
OF version = 'IBM,SF225_096'
gunzipping (0x01c00000 <- 0x00408000:0x006a8eac)...done 0x75cdf0 bytes
Finalizing device tree... using OF tree (promptr=00c39a50)
OF stdout device is: /vdevice/vty@30000000
Hypertas detected, assuming LPAR !
command line: root=/dev/sda2 xmon=on slub_debug
memory layout at init:
  alloc_bottom : 0000000002427000
  alloc_top    : 0000000008000000
  alloc_top_hi : 00000001e8000000
  rmo_top      : 0000000008000000
  ram_top      : 00000001e8000000
Looking for displays
found display   : /pci@800000020000002/pci@2,2/pci@1/display@0,
opening ... done
instantiating rtas at 0x00000000077ca000 ... done
0000000000000000 : boot cpu     0000000000000000
0000000000000002 : starting cpu hw idx 0000000000000002... done
0000000000000004 : starting cpu hw idx 0000000000000004... done
0000000000000006 : starting cpu hw idx 0000000000000006... done
copying OF device tree ...
Building dt strings...
Building dt structure...
Device tree strings 0x0000000002428000 -> 0x00000000024292fe
Device tree struct  0x000000000242a000 -> 0x000000000243f000
Calling quiesce ...
returning from prom_init
Partition configured for 8 cpus.
Starting Linux PPC64 #8 SMP Wed Apr 4 10:21:43 PDT 2007
-----------------------------------------------------
ppc64_pft_size                = 0x1b
physicalMemorySize            = 0x1e8000000
ppc64_caches.dcache_line_size = 0x80
ppc64_caches.icache_line_size = 0x80
htab_address                  = 0x0000000000000000
htab_hash_mask                = 0xfffff
-----------------------------------------------------
Linux version 2.6.21-rc5-mm4-ppc64 (root@elm3a121) (gcc version 4.1.0
(SUSE Linux)) #8 SMP Wed Apr 4 10:21:43 PDT 2007
[boot]0012 Setup Arch
No ramdisk, default root is /dev/sda2
EEH: PCI Enhanced I/O Error Handling Enabled
PPC64 nvram contains 8192 bytes
Zone PFN ranges:
  DMA             0 ->  1998848
  Normal    1998848 ->  1998848
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0:        0 ->   974848
    1:   974848 ->  1998848
[boot]0015 Setup Done
Built 2 zonelists.  Total pages: 1971520
Kernel command line: root=/dev/sda2 xmon=on slub_debug
[boot]0020 XICS Init
[boot]0021 XICS Done
PID hash table entries: 4096 (order: 12, 32768 bytes)
Console: colour dummy device 80x25
console handover: boot [udbg-1] -> real [hvc0]
Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
freeing bootmem node 0
freeing bootmem node 1
Memory: 7855400k/7995392k available (6060k kernel code, 139992k
reserved, 1236k data, 819k bss, 272k init)
SLUB V6: General Slabs=18, HW alignment=128, Processors=8, Nodes=16
Calibrating delay loop... 475.13 BogoMIPS (lpj=2375680)
Security Framework v1.0.0 initialized
Mount-cache hash table entries: 256
Processor 1 found.
Processor 2 found.
Processor 3 found.
Processor 4 found.
Processor 5 found.
Processor 6 found.
Processor 7 found.
Brought up 8 CPUs
migration_cost=0,1,2744
NET: Registered protocol family 16
IOMMU table initialized, virtual merging enabled
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
NET: Registered protocol family 2
IP route cache hash table entries: 262144 (order: 9, 2097152 bytes)
TCP established hash table entries: 524288 (order: 11, 12582912 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 524288 bind 65536)
TCP reno registered
IBM eBus Device Driver
audit: initializing netlink socket (disabled)
audit(1175707672.950:1): initialized
Total HugeTLB memory allocated, 0
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
rpaphp: RPA HOT Plug PCI Controller Driver version: 0.1
rpaphp: Slot [0001:00:02.0](PCI location=U7879.001.DQD02PW-P1-C3)
registered
rpaphp: Slot [0001:00:02.2](PCI location=U7879.001.DQD02PW-P1-C4)
registered
rpaphp: Slot [0001:00:02.4](PCI location=U7879.001.DQD02PW-P1-C5)
registered
rpaphp: Slot [0001:00:02.6](PCI location=U7879.001.DQD02PW-P1-C6)
registered
rpaphp: Slot [0002:00:02.0](PCI location=U7879.001.DQD02PW-P1-C1)
registered
rpaphp: Slot [0002:00:02.6](PCI location=U7879.001.DQD02PW-P1-C2)
registered
matroxfb: Matrox G450 detected
PInS data found at offset 31168
PInS memtype = 5
matroxfb: 640x480x8bpp (virtual: 640x26214)
matroxfb: framebuffer at 0x40170000000, mapped to 0xd000080080004000,
size 33554432
Console: switching to colour frame buffer device 80x30
fb0: MATROX frame buffer device
matroxfb_crtc2: secondary head of fb0 was registered as fb1
Kernel panic - not syncing: Couldn't register hvsi console driver

Re: 2.6.21-rc5-mm4 (SLUB powerpc)

[Christoph Lameter]

On Wed, 4 Apr 2007, Badari Pulavarty wrote:

> Next issue ? Sorry.

No problem. Could have a look at the hvsi driver and figure out what is 
failing there? What is the hvsi driver?

> Console: switching to colour frame buffer device 80x30
> fb0: MATROX frame buffer device
> matroxfb_crtc2: secondary head of fb0 was registered as fb1
> Kernel panic - not syncing: Couldn't register hvsi console driver

Framebuffer allocation failure????

Re: 2.6.21-rc5-mm4 (SLUB powerpc)

[Badari Pulavarty]

On Wed, 2007-04-04 at 10:35 -0700, Christoph Lameter wrote: 
> On Wed, 4 Apr 2007, Badari Pulavarty wrote:
> 
> > Next issue ? Sorry.
> 
> No problem. Could have a look at the hvsi driver and figure out what is 
> failing there? What is the hvsi driver?
> 
> > Console: switching to colour frame buffer device 80x30
> > fb0: MATROX frame buffer device
> > matroxfb_crtc2: secondary head of fb0 was registered as fb1
> > Kernel panic - not syncing: Couldn't register hvsi console driver
> 
> Framebuffer allocation failure????
> 

It looks like.. hvsi.c:

if (tty_register_driver(hvsi_driver))
                panic("Couldn't register hvsi console driver\n");

I added printk() in all failure cases in tty_register_driver()
and I can't reproduce the problem.

Machine tries to boot and goes further and hangs. I saw similar
hang with RSDL earlier.

Thanks,
Badari

Welcome to yaboot version 10.1.5-r625.SuSE
booted from '/pci@800000020000003/pci@2,2/pci1069,b166@1/scsi@0/sd@4,0'
Enter "help" to get some basic usage information
boot: 2621rc5mm4
Please wait, loading kernel...
Allocated 0x00400000 bytes for executable @ 0x00400000
   Elf32 kernel loaded...

zImage starting: loaded at 0x00400000 (sp: 0x01a3fe60)
Allocating 0x806af0 bytes for kernel ...
OF version = 'IBM,SF225_096'
gunzipping (0x01c00000 <- 0x00408000:0x006a4cd2)...done 0x741f90 bytes
Finalizing device tree... using OF tree (promptr=00c39a50)
OF stdout device is: /vdevice/vty@30000000
Hypertas detected, assuming LPAR !
command line: root=/dev/sda2 xmon=on slub_debug
memory layout at init:
  alloc_bottom : 000000000240b000
  alloc_top    : 0000000008000000
  alloc_top_hi : 00000001e8000000
  rmo_top      : 0000000008000000
  ram_top      : 00000001e8000000
Looking for displays
found display   : /pci@800000020000002/pci@2,2/pci@1/display@0,
opening ... done
instantiating rtas at 0x00000000077ca000 ... done
0000000000000000 : boot cpu     0000000000000000
0000000000000002 : starting cpu hw idx 0000000000000002... done
0000000000000004 : starting cpu hw idx 0000000000000004... done
0000000000000006 : starting cpu hw idx 0000000000000006... done
copying OF device tree ...
Building dt strings...
Building dt structure...
Device tree strings 0x000000000240c000 -> 0x000000000240d2fe
Device tree struct  0x000000000240e000 -> 0x0000000002423000
Calling quiesce ...
returning from prom_init
Partition configured for 8 cpus.
Starting Linux PPC64 #5 SMP Wed Apr 4 10:55:34 PDT 2007
-----------------------------------------------------
ppc64_pft_size                = 0x1b
physicalMemorySize            = 0x1e8000000
ppc64_caches.dcache_line_size = 0x80
ppc64_caches.icache_line_size = 0x80
htab_address                  = 0x0000000000000000
htab_hash_mask                = 0xfffff
-----------------------------------------------------
Linux version 2.6.21-rc4-mm1-ppc64 (root@elm3a121) (gcc version 4.1.0
(SUSE Linux)) #5 SMP Wed Apr 4 10:55:34 PDT 2007
[boot]0012 Setup Arch
No ramdisk, default root is /dev/sda2
EEH: PCI Enhanced I/O Error Handling Enabled
PPC64 nvram contains 8192 bytes
Zone PFN ranges:
  DMA             0 ->  1998848
  Normal    1998848 ->  1998848
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0:        0 ->   974848
    1:   974848 ->  1998848
[boot]0015 Setup Done
Built 2 zonelists.  Total pages: 1971520
Kernel command line: root=/dev/sda2 xmon=on slub_debug
[boot]0020 XICS Init
[boot]0021 XICS Done
PID hash table entries: 4096 (order: 12, 32768 bytes)
Console: colour dummy device 80x25
console handover: boot [udbg-1] -> real [hvc0]
Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
freeing bootmem node 0
freeing bootmem node 1
Memory: 7855772k/7995392k available (5992k kernel code, 139620k
reserved, 1224k data, 814k bss, 272k init)
Security Framework v1.0.0 initialized
Mount-cache hash table entries: 256
Processor 1 found.
Processor 2 found.
Processor 3 found.
Processor 4 found.
Processor 5 found.
Processor 6 found.
Processor 7 found.
Brought up 8 CPUs
migration_cost=0,3,25
NET: Registered protocol family 16
IOMMU table initialized, virtual merging enabled
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
NET: Registered protocol family 2
IP route cache hash table entries: 262144 (order: 9, 2097152 bytes)
TCP established hash table entries: 524288 (order: 11, 12582912 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 524288 bind 65536)
TCP reno registered
vio_bus_init: device_register returned -19
IBM eBus Device Driver
audit: initializing netlink socket (disabled)
audit(1175719341.610:1): initialized
Total HugeTLB memory allocated, 0
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Registering unionfs 2.0
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
rpaphp: RPA HOT Plug PCI Controller Driver version: 0.1
rpaphp: Slot [0001:00:02.0](PCI location=U7879.001.DQD02PW-P1-C3)
registered
rpaphp: Slot [0001:00:02.2](PCI location=U7879.001.DQD02PW-P1-C4)
registered
rpaphp: Slot [0001:00:02.4](PCI location=U7879.001.DQD02PW-P1-C5)
registered
rpaphp: Slot [0001:00:02.6](PCI location=U7879.001.DQD02PW-P1-C6)
registered
rpaphp: Slot [0002:00:02.0](PCI location=U7879.001.DQD02PW-P1-C1)
registered
rpaphp: Slot [0002:00:02.6](PCI location=U7879.001.DQD02PW-P1-C2)
registered
matroxfb: Matrox G450 detected
PInS data found at offset 31168
PInS memtype = 5
matroxfb: 640x480x8bpp (virtual: 640x26214)
matroxfb: framebuffer at 0x40170000000, mapped to 0xd000080080004000,
size 33554432
Console: switching to colour frame buffer device 80x30
fb0: MATROX frame buffer device
matroxfb_crtc2: secondary head of fb0 was registered as fb1
Generic RTC Driver v1.07
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing
disabled
pmac_zilog: 0.6 (Benjamin Herrenschmidt <benh@kernel.crashing.org>)
RAMDISK driver initialized: 16 RAM disks of 123456K size 1024 blocksize
Intel(R) PRO/1000 Network Driver - version 7.3.20-k2-NAPI
Copyright (c) 1999-2006 Intel Corporation.
e1000: 0000:d0:01.0: e1000_probe: (PCI-X:133MHz:64-bit)
00:09:6b:dd:0d:9c
e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
e1000: 0000:d0:01.1: e1000_probe: (PCI-X:133MHz:64-bit)
00:09:6b:dd:0d:9d
e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with
idebus=xx
ipr: IBM Power RAID SCSI Device Driver version: 2.3.1 (January 23, 2007)
ipr 0000:c0:01.0: Found IOA with IRQ: 83
ipr 0000:c0:01.0: Initializing IOA.
ipr 0000:c0:01.0: Starting IOA initialization sequence.
ipr 0000:c0:01.0: Adapter firmware version: 020A004E
ipr 0000:c0:01.0: IOA initialized.
scsi0 : IBM 570B Storage Adapter
scsi 0:0:15:0: Enclosure         IBM      VSBPD3E   U4SCSI 4812 PQ: 0
ANSI: 2
scsi: unknown device type 31
scsi 0:255:255:255: No Device         IBM      570B001          0150 PQ:
0 ANSI: 0
ipr 0002:c8:01.0: Found IOA with IRQ: 117
ipr 0002:c8:01.0: Starting IOA initialization sequence.
ipr 0002:c8:01.0: Adapter firmware version: 020A004E
ipr 0002:c8:01.0: IOA initialized.
scsi1 : IBM 570B Storage Adapter
scsi 1:0:4:0: Direct-Access     IBM   H0 HUS103014FL3800  RPQF PQ: 0
ANSI: 4
scsi 1:0:5:0: Direct-Access     IBM      ST373453LC       C51A PQ: 0
ANSI: 3
scsi 1:0:15:0: Enclosure         IBM      VSBPD3E   U4SCSI 4812 PQ: 0
ANSI: 2
scsi: unknown device type 31
scsi 1:255:255:255: No Device         IBM      570B001          0150 PQ:
0 ANSI: 0
sd 1:0:4:0: [sda] 286748000 512-byte hardware sectors (146815 MB)
sd 1:0:4:0: [sda] Write Protect is off
sd 1:0:4:0: [sda] Write cache: disabled, read cache: enabled, supports
DPO and FUA
sd 1:0:4:0: [sda] 286748000 512-byte hardware sectors (146815 MB)
sd 1:0:4:0: [sda] Write Protect is off
sd 1:0:4:0: [sda] Write cache: disabled, read cache: enabled, supports
DPO and FUA
 sda: sda1 sda2
sd 1:0:4:0: [sda] Attached SCSI disk
sd 1:0:5:0: [sdb] 143374000 512-byte hardware sectors (73407 MB)
sd 1:0:5:0: [sdb] Write Protect is off
sd 1:0:5:0: [sdb] Write cache: disabled, read cache: enabled, supports
DPO and FUA
sd 1:0:5:0: [sdb] 143374000 512-byte hardware sectors (73407 MB)
sd 1:0:5:0: [sdb] Write Protect is off
sd 1:0:5:0: [sdb] Write cache: disabled, read cache: enabled, supports
DPO and FUA
 sdb: sdb1 sdb2 sdb3
sd 1:0:5:0: [sdb] Attached SCSI disk
ehci_hcd 0000:c8:01.2: EHCI Host Controller
ehci_hcd 0000:c8:01.2: new USB bus registered, assigned bus number 1
ehci_hcd 0000:c8:01.2: irq 85, io mem 0x400a0002000
ehci_hcd 0000:c8:01.2: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
usb usb1: new device found, idVendor=0000, idProduct=0000
usb usb1: new device strings: Mfr=3, Product=2, SerialNumber=1
usb usb1: Product: EHCI Host Controller
usb usb1: Manufacturer: Linux 2.6.21-rc4-mm1-ppc64 ehci_hcd
usb usb1: SerialNumber: 0000:c8:01.2
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 5 ports detected
ohci_hcd 0000:c8:01.0: OHCI Host Controller
ohci_hcd 0000:c8:01.0: new USB bus registered, assigned bus number 2
ohci_hcd 0000:c8:01.0: irq 85, io mem 0x400a0001000
usb usb2: new device found, idVendor=0000, idProduct=0000
usb usb2: new device strings: Mfr=3, Product=2, SerialNumber=1
usb usb2: Product: OHCI Host Controller
usb usb2: Manufacturer: Linux 2.6.21-rc4-mm1-ppc64 ohci_hcd
usb usb2: SerialNumber: 0000:c8:01.0
usb usb2: configuration #1 chosen from 1 choice
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 3 ports detected
ohci_hcd 0000:c8:01.1: OHCI Host Controller
ohci_hcd 0000:c8:01.1: new USB bus registered, assigned bus number 3
ohci_hcd 0000:c8:01.1: irq 85, io mem 0x400a0000000
usb usb3: new device found, idVendor=0000, idProduct=0000
usb usb3: new device strings: Mfr=3, Product=2, SerialNumber=1
usb usb3: Product: OHCI Host Controller
usb usb3: Manufacturer: Linux 2.6.21-rc4-mm1-ppc64 ohci_hcd
usb usb3: SerialNumber: 0000:c8:01.1
usb usb3: configuration #1 chosen from 1 choice
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 2 ports detected
mice: PS/2 mouse device common for all mice
async_tx: api initialized (sync-only)
xor: measuring software checksumming speed
   8regs     :  6358.800 MB/sec
   8regs_prefetch:  5449.200 MB/sec
   32regs    :  6944.000 MB/sec
   32regs_prefetch:  4956.800 MB/sec
xor: using function: 32regs (6944.000 MB/sec)
usbcore: registered new interface driver hiddev
usbcore: registered new interface driver usbhid
drivers/hid/usbhid/hid-core.c: v2.6:USB HID core driver
NET: Registered protocol family 1
NET: Registered protocol family 17
NET: Registered protocol family 15
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
ReiserFS: sda2: found reiserfs format "3.6" with standard journal
ReiserFS: sda2: using ordered data mode
ReiserFS: sda2: journal params: device sda2, size 8192, journal first
block 18, max trans len 1024, max batch 900, max commit age 30, max
trans age 30
ReiserFS: sda2: checking transaction log (sda2)
ReiserFS: sda2: replayed 38 transactions in 7 seconds
ReiserFS: sda2: Using r5 hash to sort names
VFS: Mounted root (reiserfs filesystem) readonly.
Freeing unused kernel memory: 272k freed
Warning: unable to open an initial console.
ioctl32(showconsole:1020): Unknown cmd fd(0) cmd(40045432){00} arg
(ffb7cb48) on /dev/tty0
ioctl32(showconsole:1048): Unknown cmd fd(0) cmd(40045432){00} arg
(ffabdad8) on /dev/tty0
PDC20275: IDE controller at PCI slot 0002:d0:01.0
PDC20275: chipset revision 1
PDC20275: PLL input clock is 32814 kHz
PDC20275: 100% native mode on irq 119
    ide2: BM-DMA at 0x2eec00-0x2eec07, BIOS settings: hde:pio, hdf:pio
    ide3: BM-DMA at 0x2eec08-0x2eec0f, BIOS settings: hdg:pio, hdh:pio
scsi 0:0:15:0: Attached scsi generic sg0 type 13
scsi 0:255:255:255: Attached scsi generic sg1 type 31
sd 1:0:4:0: Attached scsi generic sg2 type 0
sd 1:0:5:0: Attached scsi generic sg3 type 0
scsi 1:0:15:0: Attached scsi generic sg4 type 13
scsi 1:255:255:255: Attached scsi generic sg5 type 31
hde: IBM DROM00205, ATAPI CD/DVD-ROM drive
ide2 at 0x2ee400-0x2ee407,0x2edc02 on irq 119
hde: ATAPI 24X DVD-ROM drive, 256kB Cache
Uniform CD-ROM driver Revision: 3.20
ReiserFS: sda2: Removing [653 745793 0x0 SD]..done
ReiserFS: sda2: There were 1 uncompleted unlinks/truncates. Completed
device-mapper: ioctl: 4.11.0-ioctl (2006-10-12) initialised: dm-
devel@redhat.com
ioctl32(showconsole:1924): Unknown cmd fd(0) cmd(40045432){00} arg
(fffe8ad8) on /dev/tty0
loop: loaded (max 8 devices)
ioctl32(showconsole:2098): Unknown cmd fd(0) cmd(40045432){00} arg
(ffcb0ae8) on /dev/tty0
Adding 1050616k swap on /dev/sdb2.  Priority:-1 extents:1
across:1050616k
ioctl32(showconsole:2144): Unknown cmd fd(0) cmd(40045432){00} arg
(fffebb28) on /dev/tty0
e1000: eth0: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex, Flow
Control: RX
audit(1175719380.150:2): audit_pid=2789 old=0 by auid=4294967295
ioctl32(showconsole:3015): Unknown cmd fd(0) cmd(40045432){00} arg
(ffe0ca98) on /dev/tty0

Re: 2.6.21-rc5-mm4

[Badari Pulavarty]

On Mon, 2007-04-02 at 22:47 -0700, Andrew Morton wrote:
> ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/



SLUB code paniced my x86-64 box also :(

Thanks,
Badari

Linux version 2.6.21-rc5-mm4 (root@elm3b29) (gcc version 4.1.0 (SUSE
Linux)) #1 SMP Tue Apr 3 15:45:09 PDT 2007
Command line: root=/dev/hda2 vga=0x314  selinux=0   console=tty0
console=ttyS0,38400 resume=/dev/hda1 resume=/dev/hda1  splash=silent
showopts
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009f000 (usable)
 BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000ca000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 00000000dfef0000 (usable)
 BIOS-e820: 00000000dfef0000 - 00000000dfeff000 (ACPI data)
 BIOS-e820: 00000000dfeff000 - 00000000dff00000 (ACPI NVS)
 BIOS-e820: 00000000dff00000 - 00000000e0000000 (usable)
 BIOS-e820: 00000000fec00000 - 00000000fec00400 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
 BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
 BIOS-e820: 0000000100000000 - 00000001e0000000 (usable)
end_pfn_map = 1966080
DMI 2.3 present.
ACPI: RSDP 000F6970, 0024 (r2 PTLTD )
ACPI: XSDT DFEFC625, 003C (r1 PTLTD      XSDT    6040000  LTP        0)
ACPI: FACP DFEFED02, 00F4 (r3 AMD    HAMMER    6040000 PTEC    F4240)
ACPI: DSDT DFEFC661, 262D (r1 AMD-K8  AMDACPI  6040000 MSFT  100000D)
ACPI: FACS DFEFFFC0, 0040
ACPI: SRAT DFEFEDF6, 0160 (r1 AMD    HAMMER    6040000 AMD         1)
ACPI: APIC DFEFEF56, 00AA (r1 PTLTD      APIC    6040000  LTP        0)
SRAT: PXM 0 -> APIC 0 -> Node 0
SRAT: PXM 1 -> APIC 1 -> Node 1
SRAT: PXM 2 -> APIC 2 -> Node 2
SRAT: PXM 3 -> APIC 3 -> Node 3
SRAT: Node 0 PXM 0 0-a0000
SRAT: Node 0 PXM 0 0-e0000000
SRAT: Node 0 PXM 0 0-180000000
SRAT: PXM 1 (100000000-1a0000000) overlaps with PXM 0 (0-180000000)
SRAT: SRAT not used.
Scanning NUMA topology in Northbridge 24
Number of nodes 4
Node 0 MemBase 0000000000000000 Limit 0000000180000000
Node 1 MemBase 0000000180000000 Limit 00000001a0000000
Node 2 MemBase 00000001a0000000 Limit 00000001c0000000
Node 3 MemBase 00000001c0000000 Limit 00000001e0000000
Using node hash shift of 29
Bootmem setup node 0 0000000000000000-0000000180000000
Bootmem setup node 1 0000000180000000-00000001a0000000
Bootmem setup node 2 00000001a0000000-00000001c0000000
Bootmem setup node 3 00000001c0000000-00000001e0000000
Zone PFN ranges:
  DMA             0 ->     4096
  DMA32        4096 ->  1048576
  Normal    1048576 ->  1966080
Movable zone start PFN for each node
early_node_map[7] active PFN ranges
    0:        0 ->      159
    0:      256 ->   917232
    0:   917248 ->   917504
    0:  1048576 ->  1572864
    1:  1572864 ->  1703936
    2:  1703936 ->  1835008
    3:  1835008 ->  1966080
ACPI: PM-Timer IO Port: 0x8008
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Processor #0 (Bootup-CPU)
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
Processor #1
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
Processor #2
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
Processor #3
ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 4, address 0xfec00000, GSI 0-23
ACPI: IOAPIC (id[0x05] address[0xfa3e0000] gsi_base[24])
IOAPIC[1]: apic_id 5, address 0xfa3e0000, GSI 24-27
ACPI: IOAPIC (id[0x06] address[0xfa3e1000] gsi_base[28])
IOAPIC[2]: apic_id 6, address 0xfa3e1000, GSI 28-31
ACPI: IOAPIC (id[0x07] address[0xfa3e2000] gsi_base[32])
IOAPIC[3]: apic_id 7, address 0xfa3e2000, GSI 32-35
ACPI: IOAPIC (id[0x08] address[0xfa3e4000] gsi_base[36])
IOAPIC[4]: apic_id 8, address 0xfa3e4000, GSI 36-39
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
Setting APIC routing to flat
Using ACPI (MADT) for SMP configuration information
Allocating PCI resources starting at e2000000 (gap: e0000000:1ec00000)
PERCPU: Allocating 41984 bytes of per cpu data
Built 4 zonelists.  Total pages: 1771993
Kernel command line: root=/dev/hda2 vga=0x314  selinux=0   console=tty0
console=ttyS0,38400 resume=/dev/hda1 resume=/dev/hda1  splash=silent
showopts
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 32768 bytes)
Extended CMOS year: 2000
time.c: Detected 1398.207 MHz processor.
Console: colour dummy device 80x25
Checking aperture...
CPU 0: aperture @ 0 size 32 MB
No AGP bridge found
Your BIOS doesn't leave a aperture memory hole
Please enable the IOMMU option in the BIOS setup
This costs you 64 MB of RAM
Mapping aperture over 65536 KB of RAM @ 4000000
Memory: 7021948k/7864320k available (3113k kernel code, 317632k
reserved, 1229k data, 364k init)
SLUB V6: General Slabs=18, HW alignment=128, Processors=4, Nodes=64
Calibrating delay using timer specific routine.. 2799.04 BogoMIPS
(lpj=5598083)
Security Framework v1.0.0 initialized
Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
Mount-cache hash table entries: 256
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 0/0 -> Node 0
Freeing SMP alternatives: 36k freed
ACPI: Core revision 20070126
Using local APIC timer interrupts.
result 12483999
Detected 12.483 MHz APIC timer.
Booting processor 1/4 APIC 0x1
Initializing CPU#1
Calibrating delay using timer specific routine.. 2796.59 BogoMIPS
(lpj=5593187)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 1/1 -> Node 1
Opteron MP w/ 1MB stepping 00
Booting processor 2/4 APIC 0x2
Initializing CPU#2
Calibrating delay using timer specific routine.. 2796.62 BogoMIPS
(lpj=5593246)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 2/2 -> Node 2
Opteron MP w/ 1MB stepping 00
Booting processor 3/4 APIC 0x3
Initializing CPU#3
Calibrating delay using timer specific routine.. 2796.59 BogoMIPS
(lpj=5593181)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 3/3 -> Node 3
Opteron MP w/ 1MB stepping 00
Brought up 4 CPUs
migration_cost=0,712
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: Using configuration type 1
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
0000:00:07.1: cannot adjust BAR0 (not I/O)
0000:00:07.1: cannot adjust BAR1 (not I/O)
0000:00:07.1: cannot adjust BAR2 (not I/O)
0000:00:07.1: cannot adjust BAR3 (not I/O)
ACPI: PCI Root Bridge [PCI1] (0000:08)
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 *5 10 11)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 5 *10 11)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 5 10 *11)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 5 10 *11)
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
pnp: PnP ACPI: found 14 devices
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a
report
PCI-DMA: Disabling AGP.
PCI-DMA: aperture base @ 4000000 size 65536 KB
PCI-DMA: using GART IOMMU.
PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
general protection fault: 0000 [1] <6>Time: tsc clocksource has been
installed.
SMP
last sysfs file:
CPU 0
Modules linked in:
Pid: 1, comm: swapper Not tainted 2.6.21-rc5-mm4 #1
RIP: 0010:[<ffffffff802860e1>]  [<ffffffff802860e1>] kmem_cache_alloc
+0xc1/0x1a0RSP: 0000:ffff810173f39dd0  EFLAGS: 00010006
RAX: 0000000000000000 RBX: ffff81019ff6b600 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 00000000000000d0 RDI: ffffffff8068dac0
RBP: ffff810173f39e00 R08: ffff81019ed4283a R09: 0000000000000000
R10: ffff81019ed42900 R11: 0000000000000100 R12: 0000af3500000000
R13: ffffffff8068dac0 R14: 0000000000000282 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffffffff8063e000(0000)
knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000000201000 CR4: 00000000000006e0
Process swapper (pid: 1, threadinfo ffff810173f38000, task
ffff8101c0340000)
Stack:  00000000000000d0 00000000000000d0 ffffffff80601d60
000000000000000d
 ffffffff805991c5 ffffffff805991c5 ffff810173f39e20 ffffffff80286551
 ffffffffffffffed ffff81019ed42900 ffff810173f39e50 ffffffff802a3642
Call Trace:
 [<ffffffff80286551>] __kmalloc+0x21/0x30
 [<ffffffff802a3642>] alloc_vfsmnt+0xd2/0x110
 [<ffffffff8028c4ac>] vfs_kern_mount+0x3c/0x160
 [<ffffffff8028c5e0>] kern_mount+0x10/0x20
 [<ffffffff806cef6b>] anon_inode_init+0x2b/0x130
 [<ffffffff806b2a04>] kernel_init+0x154/0x330
 [<ffffffff8020a8c8>] child_rip+0xa/0x12
 [<ffffffff806b28b0>] kernel_init+0x0/0x330
 [<ffffffff8020a8be>] child_rip+0x0/0x12


Code: 49 8b 04 c4 48 89 43 20 f0 0f ba 2b 02 8b 03 a8 01 74 4c f0
RIP  [<ffffffff802860e1>] kmem_cache_alloc+0xc1/0x1a0
 RSP <ffff810173f39dd0>
Kernel panic - not syncing: Attempted to kill init!

Re: 2.6.21-rc5-mm4

[Badari Pulavarty]

On Tue, 2007-04-03 at 14:55 -0700, Badari Pulavarty wrote:
> On Mon, 2007-04-02 at 22:47 -0700, Andrew Morton wrote:
> > ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/
> 
> 
> 
> SLUB code paniced my x86-64 box also :(

More on this panic.. line of code causing the issue is:

mm/slub.c:1133
    2df1:       49 8b 04 c4             mov    (%r12,%rax,8),%rax
    2df5:       48 89 43 20             mov    %rax,0x20(%rbx)
      
page->freelist = object[page->offset];

Hope this helps.

Thanks,
Badari

> Linux version 2.6.21-rc5-mm4 (root@elm3b29) (gcc version 4.1.0 (SUSE
> Linux)) #1 SMP Tue Apr 3 15:45:09 PDT 2007
> Command line: root=/dev/hda2 vga=0x314  selinux=0   console=tty0
> console=ttyS0,38400 resume=/dev/hda1 resume=/dev/hda1  splash=silent
> showopts
> BIOS-provided physical RAM map:
>  BIOS-e820: 0000000000000000 - 000000000009f000 (usable)
>  BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved)
>  BIOS-e820: 00000000000ca000 - 0000000000100000 (reserved)
>  BIOS-e820: 0000000000100000 - 00000000dfef0000 (usable)
>  BIOS-e820: 00000000dfef0000 - 00000000dfeff000 (ACPI data)
>  BIOS-e820: 00000000dfeff000 - 00000000dff00000 (ACPI NVS)
>  BIOS-e820: 00000000dff00000 - 00000000e0000000 (usable)
>  BIOS-e820: 00000000fec00000 - 00000000fec00400 (reserved)
>  BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
>  BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
>  BIOS-e820: 0000000100000000 - 00000001e0000000 (usable)
> end_pfn_map = 1966080
> DMI 2.3 present.
> ACPI: RSDP 000F6970, 0024 (r2 PTLTD )
> ACPI: XSDT DFEFC625, 003C (r1 PTLTD      XSDT    6040000  LTP        0)
> ACPI: FACP DFEFED02, 00F4 (r3 AMD    HAMMER    6040000 PTEC    F4240)
> ACPI: DSDT DFEFC661, 262D (r1 AMD-K8  AMDACPI  6040000 MSFT  100000D)
> ACPI: FACS DFEFFFC0, 0040
> ACPI: SRAT DFEFEDF6, 0160 (r1 AMD    HAMMER    6040000 AMD         1)
> ACPI: APIC DFEFEF56, 00AA (r1 PTLTD      APIC    6040000  LTP        0)
> SRAT: PXM 0 -> APIC 0 -> Node 0
> SRAT: PXM 1 -> APIC 1 -> Node 1
> SRAT: PXM 2 -> APIC 2 -> Node 2
> SRAT: PXM 3 -> APIC 3 -> Node 3
> SRAT: Node 0 PXM 0 0-a0000
> SRAT: Node 0 PXM 0 0-e0000000
> SRAT: Node 0 PXM 0 0-180000000
> SRAT: PXM 1 (100000000-1a0000000) overlaps with PXM 0 (0-180000000)
> SRAT: SRAT not used.
> Scanning NUMA topology in Northbridge 24
> Number of nodes 4
> Node 0 MemBase 0000000000000000 Limit 0000000180000000
> Node 1 MemBase 0000000180000000 Limit 00000001a0000000
> Node 2 MemBase 00000001a0000000 Limit 00000001c0000000
> Node 3 MemBase 00000001c0000000 Limit 00000001e0000000
> Using node hash shift of 29
> Bootmem setup node 0 0000000000000000-0000000180000000
> Bootmem setup node 1 0000000180000000-00000001a0000000
> Bootmem setup node 2 00000001a0000000-00000001c0000000
> Bootmem setup node 3 00000001c0000000-00000001e0000000
> Zone PFN ranges:
>   DMA             0 ->     4096
>   DMA32        4096 ->  1048576
>   Normal    1048576 ->  1966080
> Movable zone start PFN for each node
> early_node_map[7] active PFN ranges
>     0:        0 ->      159
>     0:      256 ->   917232
>     0:   917248 ->   917504
>     0:  1048576 ->  1572864
>     1:  1572864 ->  1703936
>     2:  1703936 ->  1835008
>     3:  1835008 ->  1966080
> ACPI: PM-Timer IO Port: 0x8008
> ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
> Processor #0 (Bootup-CPU)
> ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
> Processor #1
> ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
> Processor #2
> ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
> Processor #3
> ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
> ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
> ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
> ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
> ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
> IOAPIC[0]: apic_id 4, address 0xfec00000, GSI 0-23
> ACPI: IOAPIC (id[0x05] address[0xfa3e0000] gsi_base[24])
> IOAPIC[1]: apic_id 5, address 0xfa3e0000, GSI 24-27
> ACPI: IOAPIC (id[0x06] address[0xfa3e1000] gsi_base[28])
> IOAPIC[2]: apic_id 6, address 0xfa3e1000, GSI 28-31
> ACPI: IOAPIC (id[0x07] address[0xfa3e2000] gsi_base[32])
> IOAPIC[3]: apic_id 7, address 0xfa3e2000, GSI 32-35
> ACPI: IOAPIC (id[0x08] address[0xfa3e4000] gsi_base[36])
> IOAPIC[4]: apic_id 8, address 0xfa3e4000, GSI 36-39
> ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
> Setting APIC routing to flat
> Using ACPI (MADT) for SMP configuration information
> Allocating PCI resources starting at e2000000 (gap: e0000000:1ec00000)
> PERCPU: Allocating 41984 bytes of per cpu data
> Built 4 zonelists.  Total pages: 1771993
> Kernel command line: root=/dev/hda2 vga=0x314  selinux=0   console=tty0
> console=ttyS0,38400 resume=/dev/hda1 resume=/dev/hda1  splash=silent
> showopts
> Initializing CPU#0
> PID hash table entries: 4096 (order: 12, 32768 bytes)
> Extended CMOS year: 2000
> time.c: Detected 1398.207 MHz processor.
> Console: colour dummy device 80x25
> Checking aperture...
> CPU 0: aperture @ 0 size 32 MB
> No AGP bridge found
> Your BIOS doesn't leave a aperture memory hole
> Please enable the IOMMU option in the BIOS setup
> This costs you 64 MB of RAM
> Mapping aperture over 65536 KB of RAM @ 4000000
> Memory: 7021948k/7864320k available (3113k kernel code, 317632k
> reserved, 1229k data, 364k init)
> SLUB V6: General Slabs=18, HW alignment=128, Processors=4, Nodes=64
> Calibrating delay using timer specific routine.. 2799.04 BogoMIPS
> (lpj=5598083)
> Security Framework v1.0.0 initialized
> Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
> Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
> Mount-cache hash table entries: 256
> CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
> CPU: L2 Cache: 1024K (64 bytes/line)
> CPU 0/0 -> Node 0
> Freeing SMP alternatives: 36k freed
> ACPI: Core revision 20070126
> Using local APIC timer interrupts.
> result 12483999
> Detected 12.483 MHz APIC timer.
> Booting processor 1/4 APIC 0x1
> Initializing CPU#1
> Calibrating delay using timer specific routine.. 2796.59 BogoMIPS
> (lpj=5593187)
> CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
> CPU: L2 Cache: 1024K (64 bytes/line)
> CPU 1/1 -> Node 1
> Opteron MP w/ 1MB stepping 00
> Booting processor 2/4 APIC 0x2
> Initializing CPU#2
> Calibrating delay using timer specific routine.. 2796.62 BogoMIPS
> (lpj=5593246)
> CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
> CPU: L2 Cache: 1024K (64 bytes/line)
> CPU 2/2 -> Node 2
> Opteron MP w/ 1MB stepping 00
> Booting processor 3/4 APIC 0x3
> Initializing CPU#3
> Calibrating delay using timer specific routine.. 2796.59 BogoMIPS
> (lpj=5593181)
> CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
> CPU: L2 Cache: 1024K (64 bytes/line)
> CPU 3/3 -> Node 3
> Opteron MP w/ 1MB stepping 00
> Brought up 4 CPUs
> migration_cost=0,712
> NET: Registered protocol family 16
> ACPI: bus type pci registered
> PCI: Using configuration type 1
> ACPI: Interpreter enabled
> ACPI: Using IOAPIC for interrupt routing
> ACPI: PCI Root Bridge [PCI0] (0000:00)
> 0000:00:07.1: cannot adjust BAR0 (not I/O)
> 0000:00:07.1: cannot adjust BAR1 (not I/O)
> 0000:00:07.1: cannot adjust BAR2 (not I/O)
> 0000:00:07.1: cannot adjust BAR3 (not I/O)
> ACPI: PCI Root Bridge [PCI1] (0000:08)
> ACPI: PCI Interrupt Link [LNKA] (IRQs 3 *5 10 11)
> ACPI: PCI Interrupt Link [LNKB] (IRQs 3 5 *10 11)
> ACPI: PCI Interrupt Link [LNKC] (IRQs 3 5 10 *11)
> ACPI: PCI Interrupt Link [LNKD] (IRQs 3 5 10 *11)
> Linux Plug and Play Support v0.97 (c) Adam Belay
> pnp: PnP ACPI init
> pnp: PnP ACPI: found 14 devices
> PCI: Using ACPI for IRQ routing
> PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a
> report
> PCI-DMA: Disabling AGP.
> PCI-DMA: aperture base @ 4000000 size 65536 KB
> PCI-DMA: using GART IOMMU.
> PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
> general protection fault: 0000 [1] <6>Time: tsc clocksource has been
> installed.
> SMP
> last sysfs file:
> CPU 0
> Modules linked in:
> Pid: 1, comm: swapper Not tainted 2.6.21-rc5-mm4 #1
> RIP: 0010:[<ffffffff802860e1>]  [<ffffffff802860e1>] kmem_cache_alloc
> +0xc1/0x1a0RSP: 0000:ffff810173f39dd0  EFLAGS: 00010006
> RAX: 0000000000000000 RBX: ffff81019ff6b600 RCX: 0000000000000000
> RDX: 0000000000000000 RSI: 00000000000000d0 RDI: ffffffff8068dac0
> RBP: ffff810173f39e00 R08: ffff81019ed4283a R09: 0000000000000000
> R10: ffff81019ed42900 R11: 0000000000000100 R12: 0000af3500000000
> R13: ffffffff8068dac0 R14: 0000000000000282 R15: 0000000000000000
> FS:  0000000000000000(0000) GS:ffffffff8063e000(0000)
> knlGS:0000000000000000
> CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> CR2: 0000000000000000 CR3: 0000000000201000 CR4: 00000000000006e0
> Process swapper (pid: 1, threadinfo ffff810173f38000, task
> ffff8101c0340000)
> Stack:  00000000000000d0 00000000000000d0 ffffffff80601d60
> 000000000000000d
>  ffffffff805991c5 ffffffff805991c5 ffff810173f39e20 ffffffff80286551
>  ffffffffffffffed ffff81019ed42900 ffff810173f39e50 ffffffff802a3642
> Call Trace:
>  [<ffffffff80286551>] __kmalloc+0x21/0x30
>  [<ffffffff802a3642>] alloc_vfsmnt+0xd2/0x110
>  [<ffffffff8028c4ac>] vfs_kern_mount+0x3c/0x160
>  [<ffffffff8028c5e0>] kern_mount+0x10/0x20
>  [<ffffffff806cef6b>] anon_inode_init+0x2b/0x130
>  [<ffffffff806b2a04>] kernel_init+0x154/0x330
>  [<ffffffff8020a8c8>] child_rip+0xa/0x12
>  [<ffffffff806b28b0>] kernel_init+0x0/0x330
>  [<ffffffff8020a8be>] child_rip+0x0/0x12
> 
> 
> Code: 49 8b 04 c4 48 89 43 20 f0 0f ba 2b 02 8b 03 a8 01 74 4c f0
> RIP  [<ffffffff802860e1>] kmem_cache_alloc+0xc1/0x1a0
>  RSP <ffff810173f39dd0>
> Kernel panic - not syncing: Attempted to kill init!
> 
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: 2.6.21-rc5-mm4

[Christoph Lameter]

On Tue, 3 Apr 2007, Badari Pulavarty wrote:

> On Tue, 2007-04-03 at 14:55 -0700, Badari Pulavarty wrote:
> > On Mon, 2007-04-02 at 22:47 -0700, Andrew Morton wrote:
> > > ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/
> > 
> > 
> > 
> > SLUB code paniced my x86-64 box also :(
> 
> More on this panic.. line of code causing the issue is:
> 
> mm/slub.c:1133
>     2df1:       49 8b 04 c4             mov    (%r12,%rax,8),%rax
>     2df5:       48 89 43 20             mov    %rax,0x20(%rbx)
>       
> page->freelist = object[page->offset];
> 
> Hope this helps.

Yes. Could you boot with "slub_debug" set? Looks like a slab object may 
be getting corrupted.

x86_64 with numa emulation 4 nodes boots fine here.

Re: 2.6.21-rc5-mm4

[Badari Pulavarty]

On Tue, 2007-04-03 at 16:27 -0700, Christoph Lameter wrote:
> On Tue, 3 Apr 2007, Badari Pulavarty wrote:
> 
> > On Tue, 2007-04-03 at 14:55 -0700, Badari Pulavarty wrote:
> > > On Mon, 2007-04-02 at 22:47 -0700, Andrew Morton wrote:
> > > > ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/
> > > 
> > > 
> > > 
> > > SLUB code paniced my x86-64 box also :(
> > 
> > More on this panic.. line of code causing the issue is:
> > 
> > mm/slub.c:1133
> >     2df1:       49 8b 04 c4             mov    (%r12,%rax,8),%rax
> >     2df5:       48 89 43 20             mov    %rax,0x20(%rbx)
> >       
> > page->freelist = object[page->offset];
> > 
> > Hope this helps.
> 
> Yes. Could you boot with "slub_debug" set? Looks like a slab object may 
> be getting corrupted.
> 
> x86_64 with numa emulation 4 nodes boots fine here.

Not sure, if this helps any (slub_debug = 1) :(

Thanks,
Badari

Linux version 2.6.21-rc5-mm4 (root@elm3b29) (gcc version 4.1.0 (SUSE
Linux)) #2 SMP Tue Apr 3 17:36:21 PDT 2007
Command line: root=/dev/hda2 vga=0x314  selinux=0   console=tty0
console=ttyS0,38400 resume=/dev/hda1 resume=/dev/hda1  splash=silent
showopts
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009f000 (usable)
 BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000ca000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 00000000dfef0000 (usable)
 BIOS-e820: 00000000dfef0000 - 00000000dfeff000 (ACPI data)
 BIOS-e820: 00000000dfeff000 - 00000000dff00000 (ACPI NVS)
 BIOS-e820: 00000000dff00000 - 00000000e0000000 (usable)
 BIOS-e820: 00000000fec00000 - 00000000fec00400 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
 BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
 BIOS-e820: 0000000100000000 - 00000001e0000000 (usable)
end_pfn_map = 1966080
DMI 2.3 present.
ACPI: RSDP 000F6970, 0024 (r2 PTLTD )
ACPI: XSDT DFEFC625, 003C (r1 PTLTD      XSDT    6040000  LTP        0)
ACPI: FACP DFEFED02, 00F4 (r3 AMD    HAMMER    6040000 PTEC    F4240)
ACPI: DSDT DFEFC661, 262D (r1 AMD-K8  AMDACPI  6040000 MSFT  100000D)
ACPI: FACS DFEFFFC0, 0040
ACPI: SRAT DFEFEDF6, 0160 (r1 AMD    HAMMER    6040000 AMD         1)
ACPI: APIC DFEFEF56, 00AA (r1 PTLTD      APIC    6040000  LTP        0)
SRAT: PXM 0 -> APIC 0 -> Node 0
SRAT: PXM 1 -> APIC 1 -> Node 1
SRAT: PXM 2 -> APIC 2 -> Node 2
SRAT: PXM 3 -> APIC 3 -> Node 3
SRAT: Node 0 PXM 0 0-a0000
SRAT: Node 0 PXM 0 0-e0000000
SRAT: Node 0 PXM 0 0-180000000
SRAT: PXM 1 (100000000-1a0000000) overlaps with PXM 0 (0-180000000)
SRAT: SRAT not used.
Scanning NUMA topology in Northbridge 24
Number of nodes 4
Node 0 MemBase 0000000000000000 Limit 0000000180000000
Node 1 MemBase 0000000180000000 Limit 00000001a0000000
Node 2 MemBase 00000001a0000000 Limit 00000001c0000000
Node 3 MemBase 00000001c0000000 Limit 00000001e0000000
Using node hash shift of 29
Bootmem setup node 0 0000000000000000-0000000180000000
Bootmem setup node 1 0000000180000000-00000001a0000000
Bootmem setup node 2 00000001a0000000-00000001c0000000
Bootmem setup node 3 00000001c0000000-00000001e0000000
Zone PFN ranges:
  DMA             0 ->     4096
  DMA32        4096 ->  1048576
  Normal    1048576 ->  1966080
Movable zone start PFN for each node
early_node_map[7] active PFN ranges
    0:        0 ->      159
    0:      256 ->   917232
    0:   917248 ->   917504
    0:  1048576 ->  1572864
    1:  1572864 ->  1703936
    2:  1703936 ->  1835008
    3:  1835008 ->  1966080
ACPI: PM-Timer IO Port: 0x8008
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Processor #0 (Bootup-CPU)
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
Processor #1
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
Processor #2
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
Processor #3
ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 4, address 0xfec00000, GSI 0-23
ACPI: IOAPIC (id[0x05] address[0xfa3e0000] gsi_base[24])
IOAPIC[1]: apic_id 5, address 0xfa3e0000, GSI 24-27
ACPI: IOAPIC (id[0x06] address[0xfa3e1000] gsi_base[28])
IOAPIC[2]: apic_id 6, address 0xfa3e1000, GSI 28-31
ACPI: IOAPIC (id[0x07] address[0xfa3e2000] gsi_base[32])
IOAPIC[3]: apic_id 7, address 0xfa3e2000, GSI 32-35
ACPI: IOAPIC (id[0x08] address[0xfa3e4000] gsi_base[36])
IOAPIC[4]: apic_id 8, address 0xfa3e4000, GSI 36-39
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
Setting APIC routing to flat
Using ACPI (MADT) for SMP configuration information
Allocating PCI resources starting at e2000000 (gap: e0000000:1ec00000)
PERCPU: Allocating 41984 bytes of per cpu data
Built 4 zonelists.  Total pages: 1771993
Kernel command line: root=/dev/hda2 vga=0x314  selinux=0   console=tty0
console=ttyS0,38400 resume=/dev/hda1 resume=/dev/hda1  splash=silent
showopts
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 32768 bytes)
Extended CMOS year: 2000
time.c: Detected 1398.191 MHz processor.
Console: colour dummy device 80x25
Checking aperture...
CPU 0: aperture @ 0 size 32 MB
No AGP bridge found
Your BIOS doesn't leave a aperture memory hole
Please enable the IOMMU option in the BIOS setup
This costs you 64 MB of RAM
Mapping aperture over 65536 KB of RAM @ 4000000
Memory: 7021948k/7864320k available (3113k kernel code, 317632k
reserved, 1229k data, 364k init)
SLUB V6: General Slabs=18, HW alignment=128, Processors=4, Nodes=64
Calibrating delay using timer specific routine.. 2799.03 BogoMIPS
(lpj=5598079)
Security Framework v1.0.0 initialized
Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
Mount-cache hash table entries: 256
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 0/0 -> Node 0
Freeing SMP alternatives: 36k freed
ACPI: Core revision 20070126
Using local APIC timer interrupts.
result 12483855
Detected 12.483 MHz APIC timer.
Booting processor 1/4 APIC 0x1
Initializing CPU#1
Calibrating delay using timer specific routine.. 2796.59 BogoMIPS
(lpj=5593189)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 1/1 -> Node 1
Opteron MP w/ 1MB stepping 00
Booting processor 2/4 APIC 0x2
Initializing CPU#2
Calibrating delay using timer specific routine.. 2796.57 BogoMIPS
(lpj=5593147)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 2/2 -> Node 2
Opteron MP w/ 1MB stepping 00
Booting processor 3/4 APIC 0x3
Initializing CPU#3
Calibrating delay using timer specific routine.. 2796.56 BogoMIPS
(lpj=5593127)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 3/3 -> Node 3
Opteron MP w/ 1MB stepping 00
Brought up 4 CPUs
migration_cost=0,766
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: Using configuration type 1
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
0000:00:07.1: cannot adjust BAR0 (not I/O)
0000:00:07.1: cannot adjust BAR1 (not I/O)
0000:00:07.1: cannot adjust BAR2 (not I/O)
0000:00:07.1: cannot adjust BAR3 (not I/O)
ACPI: PCI Root Bridge [PCI1] (0000:08)
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 *5 10 11)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 5 *10 11)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 5 10 *11)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 5 10 *11)
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
pnp: PnP ACPI: found 14 devices
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a
report
PCI-DMA: Disabling AGP.
PCI-DMA: aperture base @ 4000000 size 65536 KB
PCI-DMA: using GART IOMMU.
PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
general protection fault: 0000 [1] <6>Time: tsc clocksource has been
installed.
SMP
last sysfs file:
CPU 0
Modules linked in:
Pid: 1, comm: swapper Not tainted 2.6.21-rc5-mm4 #2
RIP: 0010:[<ffffffff802860e1>]  [<ffffffff802860e1>] kmem_cache_alloc
+0xc1/0x1a0RSP: 0000:ffff8101c031bdd0  EFLAGS: 00010006
RAX: 0000000000000000 RBX: ffff81017400e900 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 00000000000000d0 RDI: ffffffff8068dac0
RBP: ffff8101c031be00 R08: ffff810000180d3a R09: 0000000000000000
R10: ffff810000180c00 R11: 0000000000000100 R12: f000ff53f000ff53
R13: ffffffff8068dac0 R14: 0000000000000282 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffffffff8063e000(0000)
knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000000201000 CR4: 00000000000006e0
Process swapper (pid: 1, threadinfo ffff8101c031a000, task
ffff810000068000)
Stack:  00000000000000d0 00000000000000d0 ffffffff80601d60
000000000000000d
 ffffffff805991c5 ffffffff805991c5 ffff8101c031be20 ffffffff80286551
 ffffffffffffffed ffff810000180c00 ffff8101c031be50 ffffffff802a3642
Call Trace:
 [<ffffffff80286551>] __kmalloc+0x21/0x30
 [<ffffffff802a3642>] alloc_vfsmnt+0xd2/0x110
 [<ffffffff8028c4ac>] vfs_kern_mount+0x3c/0x160
 [<ffffffff8028c5e0>] kern_mount+0x10/0x20
 [<ffffffff806cef6b>] anon_inode_init+0x2b/0x130
 [<ffffffff806b2a04>] kernel_init+0x154/0x330
 [<ffffffff8020a8c8>] child_rip+0xa/0x12
 [<ffffffff806b28b0>] kernel_init+0x0/0x330
 [<ffffffff8020a8be>] child_rip+0x0/0x12


Code: 49 8b 04 c4 48 89 43 20 f0 0f ba 2b 02 8b 03 a8 01 74 4c f0
RIP  [<ffffffff802860e1>] kmem_cache_alloc+0xc1/0x1a0
 RSP <ffff8101c031bdd0>
Kernel panic - not syncing: Attempted to kill init!

Re: 2.6.21-rc5-mm4

[Christoph Lameter]

On Tue, 3 Apr 2007, Badari Pulavarty wrote:

> > x86_64 with numa emulation 4 nodes boots fine here.
> 
> Not sure, if this helps any (slub_debug = 1) :(

Specify only slub_debug. The 1 is not recognized by SLUB so there were no 
debug options set.

> Linux version 2.6.21-rc5-mm4 (root@elm3b29) (gcc version 4.1.0 (SUSE
> Linux)) #2 SMP Tue Apr 3 17:36:21 PDT 2007
> Command line: root=/dev/hda2 vga=0x314  selinux=0   console=tty0
> console=ttyS0,38400 resume=/dev/hda1 resume=/dev/hda1  splash=silent
> showopts

^^^ I see no slub_debug option set.

Re: 2.6.21-rc5-mm4

[Badari Pulavarty]

On Tue, 2007-04-03 at 16:40 -0700, Christoph Lameter wrote:
> On Tue, 3 Apr 2007, Badari Pulavarty wrote:
> 
> > > x86_64 with numa emulation 4 nodes boots fine here.
> > 
> > Not sure, if this helps any (slub_debug = 1) :(
> 
> Specify only slub_debug. The 1 is not recognized by SLUB so there were no 
> debug options set.
> 
> > Linux version 2.6.21-rc5-mm4 (root@elm3b29) (gcc version 4.1.0 (SUSE
> > Linux)) #2 SMP Tue Apr 3 17:36:21 PDT 2007
> > Command line: root=/dev/hda2 vga=0x314  selinux=0   console=tty0
> > console=ttyS0,38400 resume=/dev/hda1 resume=/dev/hda1  splash=silent
> > showopts
> 
> ^^^ I see no slub_debug option set.

Hmm. booted fine with slub_debug :(

Linux version 2.6.21-rc5-mm4 (root@elm3b29) (gcc version 4.1.0 (SUSE Linux)) #2 SMP Tue Apr 3 17:36:21 PDT 2007
Command line: root=/dev/hda2 vga=0x314  slub_debug selinux=0   console=tty0 console=ttyS0,38400 resume=/dev/hda1 resume=/dev/hda1  splash=silent showopts
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009f000 (usable)
 BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000ca000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 00000000dfef0000 (usable)
 BIOS-e820: 00000000dfef0000 - 00000000dfeff000 (ACPI data)
 BIOS-e820: 00000000dfeff000 - 00000000dff00000 (ACPI NVS)
 BIOS-e820: 00000000dff00000 - 00000000e0000000 (usable)
 BIOS-e820: 00000000fec00000 - 00000000fec00400 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
 BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
 BIOS-e820: 0000000100000000 - 00000001e0000000 (usable)
end_pfn_map = 1966080
DMI 2.3 present.
ACPI: RSDP 000F6970, 0024 (r2 PTLTD )
ACPI: XSDT DFEFC625, 003C (r1 PTLTD      XSDT    6040000  LTP        0)
ACPI: FACP DFEFED02, 00F4 (r3 AMD    HAMMER    6040000 PTEC    F4240)
ACPI: DSDT DFEFC661, 262D (r1 AMD-K8  AMDACPI  6040000 MSFT  100000D)
ACPI: FACS DFEFFFC0, 0040
ACPI: SRAT DFEFEDF6, 0160 (r1 AMD    HAMMER    6040000 AMD         1)
ACPI: APIC DFEFEF56, 00AA (r1 PTLTD      APIC    6040000  LTP        0)
SRAT: PXM 0 -> APIC 0 -> Node 0
SRAT: PXM 1 -> APIC 1 -> Node 1
SRAT: PXM 2 -> APIC 2 -> Node 2
SRAT: PXM 3 -> APIC 3 -> Node 3
SRAT: Node 0 PXM 0 0-a0000
SRAT: Node 0 PXM 0 0-e0000000
SRAT: Node 0 PXM 0 0-180000000
SRAT: PXM 1 (100000000-1a0000000) overlaps with PXM 0 (0-180000000)
SRAT: SRAT not used.
Scanning NUMA topology in Northbridge 24
Number of nodes 4
Node 0 MemBase 0000000000000000 Limit 0000000180000000
Node 1 MemBase 0000000180000000 Limit 00000001a0000000
Node 2 MemBase 00000001a0000000 Limit 00000001c0000000
Node 3 MemBase 00000001c0000000 Limit 00000001e0000000
Using node hash shift of 29
Bootmem setup node 0 0000000000000000-0000000180000000
Bootmem setup node 1 0000000180000000-00000001a0000000
Bootmem setup node 2 00000001a0000000-00000001c0000000
Bootmem setup node 3 00000001c0000000-00000001e0000000
Zone PFN ranges:
  DMA             0 ->     4096
  DMA32        4096 ->  1048576
  Normal    1048576 ->  1966080
Movable zone start PFN for each node
early_node_map[7] active PFN ranges
    0:        0 ->      159
    0:      256 ->   917232
    0:   917248 ->   917504
    0:  1048576 ->  1572864
    1:  1572864 ->  1703936
    2:  1703936 ->  1835008
    3:  1835008 ->  1966080
ACPI: PM-Timer IO Port: 0x8008
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Processor #0 (Bootup-CPU)
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
Processor #1
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
Processor #2
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
Processor #3
ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 4, address 0xfec00000, GSI 0-23
ACPI: IOAPIC (id[0x05] address[0xfa3e0000] gsi_base[24])
IOAPIC[1]: apic_id 5, address 0xfa3e0000, GSI 24-27
ACPI: IOAPIC (id[0x06] address[0xfa3e1000] gsi_base[28])
IOAPIC[2]: apic_id 6, address 0xfa3e1000, GSI 28-31
ACPI: IOAPIC (id[0x07] address[0xfa3e2000] gsi_base[32])
IOAPIC[3]: apic_id 7, address 0xfa3e2000, GSI 32-35
ACPI: IOAPIC (id[0x08] address[0xfa3e4000] gsi_base[36])
IOAPIC[4]: apic_id 8, address 0xfa3e4000, GSI 36-39
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
Setting APIC routing to flat
Using ACPI (MADT) for SMP configuration information
Allocating PCI resources starting at e2000000 (gap: e0000000:1ec00000)
PERCPU: Allocating 41984 bytes of per cpu data
Built 4 zonelists.  Total pages: 1771993
Kernel command line: root=/dev/hda2 vga=0x314  slub_debug selinux=0   console=tty0 console=ttyS0,38400 resume=/dev/hda1 resume=/dev/hda1  splash=silent showoptsInitializing CPU#0
PID hash table entries: 4096 (order: 12, 32768 bytes)
Extended CMOS year: 2000
time.c: Detected 1398.208 MHz processor.
Console: colour dummy device 80x25
Checking aperture...
CPU 0: aperture @ 0 size 32 MB
No AGP bridge found
Your BIOS doesn't leave a aperture memory hole
Please enable the IOMMU option in the BIOS setup
This costs you 64 MB of RAM
Mapping aperture over 65536 KB of RAM @ 4000000
Memory: 7021948k/7864320k available (3113k kernel code, 317632k reserved, 1229k data, 364k init)
SLUB V6: General Slabs=18, HW alignment=128, Processors=4, Nodes=64
Calibrating delay using timer specific routine.. 2798.98 BogoMIPS (lpj=5597964)
Security Framework v1.0.0 initialized
Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
Mount-cache hash table entries: 256
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 0/0 -> Node 0
Freeing SMP alternatives: 36k freed
ACPI: Core revision 20070126
Using local APIC timer interrupts.
result 12484009
Detected 12.484 MHz APIC timer.
Booting processor 1/4 APIC 0x1
Initializing CPU#1
Calibrating delay using timer specific routine.. 2796.59 BogoMIPS (lpj=5593189)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 1/1 -> Node 1
Opteron MP w/ 1MB stepping 00
Booting processor 2/4 APIC 0x2
Initializing CPU#2
Calibrating delay using timer specific routine.. 2796.59 BogoMIPS (lpj=5593186)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 2/2 -> Node 2
Opteron MP w/ 1MB stepping 00
Booting processor 3/4 APIC 0x3
Initializing CPU#3
Calibrating delay using timer specific routine.. 2796.59 BogoMIPS (lpj=5593188)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 3/3 -> Node 3
Opteron MP w/ 1MB stepping 00
Brought up 4 CPUs
migration_cost=0,764
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: Using configuration type 1
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
0000:00:07.1: cannot adjust BAR0 (not I/O)
0000:00:07.1: cannot adjust BAR1 (not I/O)
0000:00:07.1: cannot adjust BAR2 (not I/O)
0000:00:07.1: cannot adjust BAR3 (not I/O)
ACPI: PCI Root Bridge [PCI1] (0000:08)
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 *5 10 11)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 5 *10 11)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 5 10 *11)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 5 10 *11)
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
pnp: PnP ACPI: found 14 devices
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report
PCI-DMA: Disabling AGP.
PCI-DMA: aperture base @ 4000000 size 65536 KB
PCI-DMA: using GART IOMMU.
PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
pnp: 00:05: ioport range 0x4d0-0x4d1 has been reserved
Time: tsc clocksource has been installed.
pnp: 00:05: ioport range 0x1100-0x117f has been reserved
pnp: 00:05: ioport range 0x1180-0x11ff has been reserved
pnp: 00:06: iomem range 0xe0000-0xfffff could not be reserved
pnp: 00:06: iomem range 0xc0000-0xc87ff has been reserved
pnp: 00:06: iomem range 0xfec00000-0xfec00fff could not be reserved
pnp: 00:06: iomem range 0xffc00000-0xfff7ffff has been reserved
PCI: Bridge: 0000:00:06.0
Time: acpi_pm clocksource has been installed.
  IO window: 2000-2fff
  MEM window: fa000000-fa0fffff
  PREFETCH window: e2000000-e20fffff
PCI: Bridge: 0000:09:01.0
  IO window: disabled.
  MEM window: fa400000-faffffff
  PREFETCH window: fc000000-fdffffff
PCI: Bridge: 0000:08:01.0
  IO window: disabled.
  MEM window: fa400000-faffffff
  PREFETCH window: fc000000-fdffffff
PCI: Bridge: 0000:08:02.0
  IO window: 3000-3fff
  MEM window: fb000000-fb0fffff
  PREFETCH window: e2100000-e21fffff
PCI: Bridge: 0000:08:03.0
  IO window: disabled.
  MEM window: disabled.
  PREFETCH window: disabled.
PCI: Bridge: 0000:08:04.0
  IO window: 4000-4fff
  MEM window: fb100000-fb1fffff
  PREFETCH window: e2200000-e22fffff
ACPI: PCI Interrupt 0000:08:04.0[A] -> GSI 36 (level, low) -> IRQ 36
NET: Registered protocol family 2
IP route cache hash table entries: 262144 (order: 9, 2097152 bytes)
TCP established hash table entries: 1048576 (order: 12, 25165824 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 1048576 bind 65536)
TCP reno registered
Total HugeTLB memory allocated, 0
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
vesafb: framebuffer at 0xfc000000, mapped to 0xffffc20002780000, using 1875k, total 16384k
vesafb: mode is 800x600x16, linelength=1600, pages=16
vesafb: scrolling: redraw
vesafb: Truecolor: size=0:5:6:5, shift=0:11:5:0
Console: switching to colour frame buffer device 100x37
fb0: VESA VGA frame buffer device
Real Time Clock Driver v1.12ac
Non-volatile memory driver v1.2
AMD768 RNG detected
Linux agpgart interface v0.102 (c) Dave Jones
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing disabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
00:09: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
00:0a: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
RAMDISK driver initialized: 16 RAM disks of 128000K size 1024 blocksize
loop: module loaded
tg3.c:v3.75 (March 23, 2007)
ACPI: PCI Interrupt 0000:19:02.0[A] -> GSI 38 (level, low) -> IRQ 38
eth0: Tigon3 [partno(3C996B-T) rev 0105 PHY(5701)] (PCI:66MHz:64-bit) 10/100/1000Base-T Ethernet 00:04:76:f0:f9:aa
eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] WireSpeed[1] TSOcap[0]
eth0: dma_rwctrl[76ff000f] dma_mask[64-bit]
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
AMD8111: IDE controller at PCI slot 0000:00:07.1
AMD8111: chipset revision 3
AMD8111: not 100% native mode: will probe irqs later
AMD8111: 0000:00:07.1 (rev 03) UDMA133 controller
    ide0: BM-DMA at 0x1020-0x1027, BIOS settings: hda:DMA, hdb:pio
    ide1: BM-DMA at 0x1028-0x102f, BIOS settings: hdc:DMA, hdd:pio
hda: IC35L080AVVA07-0, ATA DISK drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
hdc: TOSHIBA DVD-ROM SD-M1612, ATAPI CD/DVD-ROM drive
ide1 at 0x170-0x177,0x376 on irq 15
hda: max request size: 128KiB
hda: 160836480 sectors (82348 MB) w/1863KiB Cache, CHS=65535/16/63, UDMA(100)
hda: cache flushes supported
 hda: hda1 hda2
hdc: ATAPI 48X DVD-ROM drive, 512kB Cache, UDMA(33)
Uniform CD-ROM driver Revision: 3.20
ide-floppy driver 0.99.newide
PNP: PS/2 Controller [PNP0303:PS2K,PNP0f13:PS2M] at 0x60,0x64 irq 1,12
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
input: AT Translated Set 2 keyboard as /class/input/input0
input: PC Speaker as /class/input/input1
input: PS/2 Generic Mouse as /class/input/input2
async_tx: api initialized (sync-only)
xor: automatically using best checksumming function: generic_sse
   generic_sse:  4488.000 MB/sec
xor: using function: generic_sse (4488.000 MB/sec)
TCP cubic registered
NET: Registered protocol family 1
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
ReiserFS: hda2: found reiserfs format "3.6" with standard journal
ReiserFS: hda2: using ordered data mode
ReiserFS: hda2: journal params: device hda2, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30
ReiserFS: hda2: checking transaction log (hda2)
ReiserFS: hda2: Using r5 hash to sort names
VFS: Mounted root (reiserfs filesystem) readonly.
Freeing unused kernel memory: 364k freed
INIT: version 2.86 booting
System Boot Control: Running /etc/init.d/boot
Mounting procfs at /proc                                             done
Mounting sysfs at /sys                                               done
Mounting tmpfs at /dev                                               done
Initializing /dev                                                    done
Mounting devpts at /dev/pts                                          done
parport_pc 00:0b: reported by Plug and Play ACPI
parport0: PC-style at 0x378 (0x778), irq 7, dma 3 [PCSPP,TRISTATE,COMPAT,EPP,ECP,DMA]
lp0: using parport0 (interrupt-driven).
Starting udevd Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077
                                                                     done
Activating swap-devices in /etc/fstab...                             done
showconsole: Warning: the ioctl TIOCGDEV is not known by the kernel
/bin/mknod -m600 /dev/shm/root b 3 2
Checking root file system...
fsck 1.40-WIP (02-Oct-2006)
[/sbin/fsck.reiserfs (1) -- /dev/shm/root] fsck.reiserfs -a /dev/shm/root
Reiserfs super block in block 16 on 0x302 of format 3.6 with standard journal
Blocks (total/free): 19842352/8198650 by 4096 bytes
Filesystem is clean
Filesystem seems mounted read-only. Skipping journal replay.
Checking internal tree..finished                                     done
Loading required kernel modules                                      done
device-mapper: ioctl: 4.11.0-ioctl (2006-10-12) initialised: dm-devel@redhat.comActivating device mapper...
                                                                     done
Starting MD Raid                                                     done
Waiting for udev to settle...
Scanning for LVM volume groups...
  Reading all physical volumes.  This may take a while...
  No volume groups found
  No volume groups found
Activating LVM volume groups...
  No volume groups found
                                                                     done
showconsole: Warning: the ioctl TIOCGDEV is not known by the kernel
Checking file systems...
fsck 1.40-WIP (02-Oct-2006)
Checking all file systems.                                           done
                                                                     done
Mounting local file systems...
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
udev on /dev type tmpfs (rw)
devpts on /dev/pts type devpts (rw,mode=0620,gid=5)
nothing was mounted                                                  done
Setting up the hardware clock                                        done
FATAL: Module apparmor not found.
Loading AppArmor module                                              failed
- could not start AppArmor                                           failed
Creating /var/log/boot.msg                                           done
showconsole: Warning: the ioctl TIOCGDEV is not known by the kernel
Setting up hostname 'elm3b29'                                        done
Setting up loopback interface     lo
    lo        IP address: 127.0.0.1/8
                                                                     done
Activating remaining swap-devices in /etc/fstab...
Adding 1048784k swap on /dev/hda1.  Priority:-1 extents:1 across:1048done
Setting current sysctl status from /etc/sysctl.conf
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.conf.all.rp_filter = 1
                                                                     done
Enabling syn flood protection                                        done
Disabling IP forwarding                                              done
                                                                     done
System Boot Control: The system has been                             set up
Skipped features:                                             boot.cycle
System Boot Control: Running /etc/init.d/boot.local                  done
INIT: Entering runlevel: 3
blogd: Warning: the ioctl TIOCGDEV is not known by the kernel
Boot logging started on /dev/ttyS0(/dev/console) at Tue Apr  3 17:52:03 2007
Master Resource Control: previous runlevel: N, switching to runlevel:3
FATAL: Module apparmor not found.
Loading AppArmor module                                              failed
- could not start AppArmor                                           failed
Initializing random number generator                                 done
Starting resource manager                                            done
Starting D-BUS daemon                                                done
input: Power Button (FF) as /class/input/input3                      done

ACPI: Power Button (FF) [PWRF]
input: Sleep Button (FF) as /class/input/input4
ACPI: Sleep Button (FF) [SLPF]
input: Power Button (CM) as /class/input/input5
ACPI: Power Button (CM) [PWRB]
[acpid]: starting up

[acpid]: 0 rules loaded

acpid: loading ACPI modules ( ac battery button fan processor thermaldone
acpid: will skip unsupported modules
acpid: loading ACPI laptop modules ( )                               done
Starting acpid                                                       done
Starting EWLM daemon
Starting HAL daemon                                                  done

Re: 2.6.21-rc5-mm4

[Christoph Lameter]

On Tue, 3 Apr 2007, Badari Pulavarty wrote:

> Hmm. booted fine with slub_debug :(

Try to selectively disable debug options... if you got the 
time...

F.e. Try with sanity checks only

slub_debug=F

then with redzoning

slub_debug=FZ

and do one with poisoning

slub_debug=FP

Maybe slub_debug avoided a race condition.

Re: 2.6.21-rc5-mm4

[Badari Pulavarty]

[-- Attachment #1: Type: text/plain, Size: 312 bytes --]

On Tue, 2007-04-03 at 16:55 -0700, Christoph Lameter wrote:
> On Tue, 3 Apr 2007, Badari Pulavarty wrote:
> 
> > Hmm. booted fine with slub_debug :(
> 
> Try to selectively disable debug options... if you got the 
> time...
> 
> F.e. Try with sanity checks only
> 
> slub_debug=F

slub_debug=F got something. 



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: slub_debug=F --]
[-- Type: text/plain; name="slub_debug=F"; charset=utf-8, Size: 87576 bytes --]

Linux version 2.6.21-rc5-mm4 (root@elm3b29) (gcc version 4.1.0 (SUSE Linux)) #2 SMP Tue Apr 3 17:36:21 PDT 2007
Command line: root=/dev/hda2 vga=0x314  slub_debug=F selinux=0   console=tty0 console=ttyS0,38400 resume=/dev/hda1 resume=/dev/hda1  splash=silent showopts
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009f000 (usable)
 BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000ca000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 00000000dfef0000 (usable)
 BIOS-e820: 00000000dfef0000 - 00000000dfeff000 (ACPI data)
 BIOS-e820: 00000000dfeff000 - 00000000dff00000 (ACPI NVS)
 BIOS-e820: 00000000dff00000 - 00000000e0000000 (usable)
 BIOS-e820: 00000000fec00000 - 00000000fec00400 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
 BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
 BIOS-e820: 0000000100000000 - 00000001e0000000 (usable)
end_pfn_map = 1966080
DMI 2.3 present.
ACPI: RSDP 000F6970, 0024 (r2 PTLTD )
ACPI: XSDT DFEFC625, 003C (r1 PTLTD  	 XSDT    6040000  LTP        0)
ACPI: FACP DFEFED02, 00F4 (r3 AMD    HAMMER    6040000 PTEC    F4240)
ACPI: DSDT DFEFC661, 262D (r1 AMD-K8  AMDACPI  6040000 MSFT  100000D)
ACPI: FACS DFEFFFC0, 0040
ACPI: SRAT DFEFEDF6, 0160 (r1 AMD    HAMMER    6040000 AMD         1)
ACPI: APIC DFEFEF56, 00AA (r1 PTLTD  	 APIC    6040000  LTP        0)
SRAT: PXM 0 -> APIC 0 -> Node 0
SRAT: PXM 1 -> APIC 1 -> Node 1
SRAT: PXM 2 -> APIC 2 -> Node 2
SRAT: PXM 3 -> APIC 3 -> Node 3
SRAT: Node 0 PXM 0 0-a0000
SRAT: Node 0 PXM 0 0-e0000000
SRAT: Node 0 PXM 0 0-180000000
SRAT: PXM 1 (100000000-1a0000000) overlaps with PXM 0 (0-180000000)
SRAT: SRAT not used.
Scanning NUMA topology in Northbridge 24
Number of nodes 4
Node 0 MemBase 0000000000000000 Limit 0000000180000000
Node 1 MemBase 0000000180000000 Limit 00000001a0000000
Node 2 MemBase 00000001a0000000 Limit 00000001c0000000
Node 3 MemBase 00000001c0000000 Limit 00000001e0000000
Using node hash shift of 29
Bootmem setup node 0 0000000000000000-0000000180000000
Bootmem setup node 1 0000000180000000-00000001a0000000
Bootmem setup node 2 00000001a0000000-00000001c0000000
Bootmem setup node 3 00000001c0000000-00000001e0000000
Zone PFN ranges:
  DMA             0 ->     4096
  DMA32        4096 ->  1048576
  Normal    1048576 ->  1966080
Movable zone start PFN for each node
early_node_map[7] active PFN ranges
    0:        0 ->      159
    0:      256 ->   917232
    0:   917248 ->   917504
    0:  1048576 ->  1572864
    1:  1572864 ->  1703936
    2:  1703936 ->  1835008
    3:  1835008 ->  1966080
ACPI: PM-Timer IO Port: 0x8008
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Processor #0 (Bootup-CPU)
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
Processor #1
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
Processor #2
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
Processor #3
ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 4, address 0xfec00000, GSI 0-23
ACPI: IOAPIC (id[0x05] address[0xfa3e0000] gsi_base[24])
IOAPIC[1]: apic_id 5, address 0xfa3e0000, GSI 24-27
ACPI: IOAPIC (id[0x06] address[0xfa3e1000] gsi_base[28])
IOAPIC[2]: apic_id 6, address 0xfa3e1000, GSI 28-31
ACPI: IOAPIC (id[0x07] address[0xfa3e2000] gsi_base[32])
IOAPIC[3]: apic_id 7, address 0xfa3e2000, GSI 32-35
ACPI: IOAPIC (id[0x08] address[0xfa3e4000] gsi_base[36])
IOAPIC[4]: apic_id 8, address 0xfa3e4000, GSI 36-39
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
Setting APIC routing to flat
Using ACPI (MADT) for SMP configuration information
Allocating PCI resources starting at e2000000 (gap: e0000000:1ec00000)
PERCPU: Allocating 41984 bytes of per cpu data
Built 4 zonelists.  Total pages: 1771993
Kernel command line: root=/dev/hda2 vga=0x314  slub_debug=F selinux=0   console=tty0 console=ttyS0,38400 resume=/dev/hda1 resume=/dev/hda1  splash=silent showopts
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 32768 bytes)
Extended CMOS year: 2000
time.c: Detected 1398.218 MHz processor.
Console: colour dummy device 80x25
Checking aperture...
CPU 0: aperture @ 0 size 32 MB
No AGP bridge found
Your BIOS doesn't leave a aperture memory hole
Please enable the IOMMU option in the BIOS setup
This costs you 64 MB of RAM
Mapping aperture over 65536 KB of RAM @ 4000000
Memory: 7021948k/7864320k available (3113k kernel code, 317632k reserved, 1229k data, 364k init)
SLUB V6: General Slabs=18, HW alignment=128, Processors=4, Nodes=64
Calibrating delay using timer specific routine.. 2798.99 BogoMIPS (lpj=5597989)
Security Framework v1.0.0 initialized
Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
Mount-cache hash table entries: 256
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 0/0 -> Node 0
Freeing SMP alternatives: 36k freed
ACPI: Core revision 20070126
Using local APIC timer interrupts.
result 12484097
Detected 12.484 MHz APIC timer.
Booting processor 1/4 APIC 0x1
Initializing CPU#1
Calibrating delay using timer specific routine.. 2796.59 BogoMIPS (lpj=5593188)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 1/1 -> Node 1
Opteron MP w/ 1MB stepping 00
Booting processor 2/4 APIC 0x2
Initializing CPU#2
Calibrating delay using timer specific routine.. 2796.59 BogoMIPS (lpj=5593187)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 2/2 -> Node 2
Opteron MP w/ 1MB stepping 00
Booting processor 3/4 APIC 0x3
Initializing CPU#3
Calibrating delay using timer specific routine.. 2796.59 BogoMIPS (lpj=5593189)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 3/3 -> Node 3
Opteron MP w/ 1MB stepping 00
Brought up 4 CPUs
migration_cost=0,769
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: Using configuration type 1
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
0000:00:07.1: cannot adjust BAR0 (not I/O)
0000:00:07.1: cannot adjust BAR1 (not I/O)
0000:00:07.1: cannot adjust BAR2 (not I/O)
0000:00:07.1: cannot adjust BAR3 (not I/O)
ACPI: PCI Root Bridge [PCI1] (0000:08)
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 *5 10 11)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 5 *10 11)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 5 10 *11)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 5 10 *11)
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
pnp: PnP ACPI: found 14 devices
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report
*** SLUB: Freepointer corrupt in kmalloc-16@0xffff8100001e22a0 Slab 0xffff81017400f100
    offset=672 flags=0x00c7 inuse=42 freelist=0xffff8100001e22a0
  Bytes b4 0xffff8100001e2290:  a0 22 1e 00 00 00 00 00 00 00 00 00 00 00 00 00 ."..............
    Object 0xffff8100001e22a0:  00 00 00 00 00 81 ff ff 00 00 00 00 00 00 00 00 ......ÿÿ........
FreePointer 0xffff8100001e22a0 -> 0xffff810000000000

Call Trace:
 [<ffffffff802840e5>] object_err+0x105/0x1b0
 [<ffffffff80284485>] check_object+0x1b5/0x1d0
 [<ffffffff80284ad4>] alloc_object_checks+0x64/0x110
 [<ffffffff8028611c>] kmem_cache_alloc+0xfc/0x1a0
 [<ffffffff802dcb27>] sysfs_create_link+0xb7/0x160
 [<ffffffff8024f431>] module_add_driver+0x41/0xd0
 [<ffffffff8040241e>] bus_add_driver+0xce/0x1d0
 [<ffffffff8040324d>] driver_register+0x5d/0x90
 [<ffffffff803a16a8>] __pci_register_driver+0x68/0xb0
 [<ffffffff806d6066>] agp_amd64_init+0x36/0xe0
 [<ffffffff806c3ad6>] gart_iommu_init+0x4c6/0x560
 [<ffffffff80227a8e>] __wake_up+0x4e/0x70
 [<ffffffff80495eb0>] genl_rcv+0x0/0x70
 [<ffffffff8049432c>] netlink_kernel_create+0x14c/0x160
 [<ffffffff80495cd0>] genl_unlock+0x10/0x40
 [<ffffffff806bc8de>] pci_iommu_init+0xe/0x20
 [<ffffffff806b2a04>] kernel_init+0x154/0x330
 [<ffffffff8020a8c8>] child_rip+0xa/0x12
 [<ffffffff806b28b0>] kernel_init+0x0/0x330
 [<ffffffff8020a8be>] child_rip+0x0/0x12

PCI-DMA: Disabling AGP.
PCI-DMA: aperture base @ 4000000 size 65536 KB
PCI-DMA: using GART IOMMU.
PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
pnp: 00:05: ioport range 0x4d0-0x4d1 has been reserved
Time: tsc clocksource has been installed.
pnp: 00:05: ioport range 0x1100-0x117f has been reserved
pnp: 00:05: ioport range 0x1180-0x11ff has been reserved
pnp: 00:06: iomem range 0xe0000-0xfffff could not be reserved
pnp: 00:06: iomem range 0xc0000-0xc87ff has been reserved
pnp: 00:06: iomem range 0xfec00000-0xfec00fff could not be reserved
pnp: 00:06: iomem range 0xffc00000-0xfff7ffff has been reserved
PCI: Bridge: 0000:00:06.0
Time: acpi_pm clocksource has been installed.
  IO window: 2000-2fff
  MEM window: fa000000-fa0fffff
  PREFETCH window: e2000000-e20fffff
PCI: Bridge: 0000:09:01.0
  IO window: disabled.
  MEM window: fa400000-faffffff
  PREFETCH window: fc000000-fdffffff
PCI: Bridge: 0000:08:01.0
  IO window: disabled.
  MEM window: fa400000-faffffff
  PREFETCH window: fc000000-fdffffff
PCI: Bridge: 0000:08:02.0
  IO window: 3000-3fff
  MEM window: fb000000-fb0fffff
  PREFETCH window: e2100000-e21fffff
PCI: Bridge: 0000:08:03.0
  IO window: disabled.
  MEM window: disabled.
  PREFETCH window: disabled.
PCI: Bridge: 0000:08:04.0
  IO window: 4000-4fff
  MEM window: fb100000-fb1fffff
  PREFETCH window: e2200000-e22fffff
ACPI: PCI Interrupt 0000:08:04.0[A] -> GSI 36 (level, low) -> IRQ 36
NET: Registered protocol family 2
IP route cache hash table entries: 262144 (order: 9, 2097152 bytes)
TCP established hash table entries: 1048576 (order: 12, 25165824 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 1048576 bind 65536)
TCP reno registered
Total HugeTLB memory allocated, 0
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
vesafb: framebuffer at 0xfc000000, mapped to 0xffffc20002780000, using 1875k, total 16384k
vesafb: mode is 800x600x16, linelength=1600, pages=16
vesafb: scrolling: redraw
vesafb: Truecolor: size=0:5:6:5, shift=0:11:5:0
Console: switching to colour frame buffer device 100x37
fb0: VESA VGA frame buffer device
Real Time Clock Driver v1.12ac
Non-volatile memory driver v1.2
AMD768 RNG detected
Linux agpgart interface v0.102 (c) Dave Jones
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing disabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
00:09: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
00:0a: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
RAMDISK driver initialized: 16 RAM disks of 128000K size 1024 blocksize
loop: module loaded
tg3.c:v3.75 (March 23, 2007)
ACPI: PCI Interrupt 0000:19:02.0[A] -> GSI 38 (level, low) -> IRQ 38
eth0: Tigon3 [partno(3C996B-T) rev 0105 PHY(5701)] (PCI:66MHz:64-bit) 10/100/1000Base-T Ethernet 00:04:76:f0:f9:aa
eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] WireSpeed[1] TSOcap[0]
eth0: dma_rwctrl[76ff000f] dma_mask[64-bit]
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
AMD8111: IDE controller at PCI slot 0000:00:07.1
AMD8111: chipset revision 3
AMD8111: not 100% native mode: will probe irqs later
AMD8111: 0000:00:07.1 (rev 03) UDMA133 controller
    ide0: BM-DMA at 0x1020-0x1027, BIOS settings: hda:DMA, hdb:pio
    ide1: BM-DMA at 0x1028-0x102f, BIOS settings: hdc:DMA, hdd:pio
hda: IC35L080AVVA07-0, ATA DISK drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
hdc: TOSHIBA DVD-ROM SD-M1612, ATAPI CD/DVD-ROM drive
ide1 at 0x170-0x177,0x376 on irq 15
hda: max request size: 128KiB
hda: 160836480 sectors (82348 MB) w/1863KiB Cache, CHS=65535/16/63, UDMA(100)
hda: cache flushes supported
 hda: hda1 hda2
hdc: ATAPI 48X DVD-ROM drive, 512kB Cache, UDMA(33)
Uniform CD-ROM driver Revision: 3.20
ide-floppy driver 0.99.newide
PNP: PS/2 Controller [PNP0303:PS2K,PNP0f13:PS2M] at 0x60,0x64 irq 1,12
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
input: AT Translated Set 2 keyboard as /class/input/input0
input: PC Speaker as /class/input/input1
input: PS/2 Generic Mouse as /class/input/input2
async_tx: api initialized (sync-only)
xor: automatically using best checksumming function: generic_sse
   generic_sse:  4489.000 MB/sec
xor: using function: generic_sse (4489.000 MB/sec)
TCP cubic registered
NET: Registered protocol family 1
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
ReiserFS: hda2: found reiserfs format "3.6" with standard journal
ReiserFS: hda2: using ordered data mode
ReiserFS: hda2: journal params: device hda2, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30
ReiserFS: hda2: checking transaction log (hda2)
ReiserFS: hda2: Using r5 hash to sort names
VFS: Mounted root (reiserfs filesystem) readonly.
Freeing unused kernel memory: 364k freed
INIT: version 2.86 booting
System Boot Control: Running /etc/init.d/boot
Mounting procfs at /proc^[7^[[?25l^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h
Mounting sysfs at /sys^[7^[[?25l^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h
Mounting tmpfs at /dev^[7^[[?25l^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h
Initializing /dev^[7^[[?25l^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h
Mounting devpts at /dev/pts^[7^[[?25l^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h
parport_pc 00:0b: reported by Plug and Play ACPI
parport0: PC-style at 0x378 (0x778), irq 7, dma 3 [PCSPP,TRISTATE,COMPAT,EPP,ECP,DMA]
lp0: using parport0 (interrupt-driven).
Starting udevd hwup invoked oom-killer: gfp_mask=0xd1, order=0, oomkilladj=-17

Call Trace:
hwup invoked oom-killer: gfp_mask=0x40d1, order=2, oomkilladj=-17
 [<ffffffff80230e35>] printk_ratelimit+0x15/0x20
 [<ffffffff802628d4>] out_of_memory+0x254/0x2b0
 [<ffffffff802647fc>] __alloc_pages+0x27c/0x3d0
 [<ffffffff8025f363>] find_lock_page+0x33/0xb0
 [<ffffffff8027ea8a>] alloc_pages_current+0x5a/0x90
 [<ffffffff80285014>] new_slab+0x244/0x290
 [<ffffffff80286154>] kmem_cache_alloc+0x134/0x1a0
 [<ffffffff8028b6b9>] get_empty_filp+0x39/0x160
 [<ffffffff8029620a>] __path_lookup_intent_open+0x3a/0xc0
 [<ffffffff8029631c>] path_lookup_open+0xc/0x10
 [<ffffffff80296430>] open_namei+0x70/0x6d0
 [<ffffffff80291f8c>] path_release+0x2c/0x40
 [<ffffffff8028dd78>] cp_new_stat+0xf8/0x120
 [<ffffffff80283ec9>] check_valid_pointer+0x29/0x90
 [<ffffffff80288908>] do_filp_open+0x28/0x50
 [<ffffffff80284b09>] alloc_object_checks+0x99/0x110
 [<ffffffff80288576>] get_unused_fd+0x76/0x110
 [<ffffffff8028898a>] do_sys_open+0x5a/0xf0
 [<ffffffff80288a4b>] sys_open+0x1b/0x20
 [<ffffffff80209aae>] system_call+0x7e/0x83

Mem-info:
Node 0 DMA per-cpu:
CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    2: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    3: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 0 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 1 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 2 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 3 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Active:5861 inactive:1292 dirty:0 writeback:0 unstable:0
 free:1733706 slab:2596 mapped:658 pagetables:690 bounce:0
Node 0 DMA free:120kB min:12kB low:12kB high:16kB active:0kB inactive:0kB present:9636kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 3440 5424 5424
Node 0 DMA32 free:3587656kB min:5292kB low:6612kB high:7936kB active:0kB inactive:0kB present:3523008kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1984 1984
Node 0 Normal free:1884664kB min:3052kB low:3812kB high:4576kB active:3388kB inactive:32kB present:2031616kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 1 Normal free:491864kB min:760kB low:948kB high:1140kB active:3568kB inactive:164kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 2 Normal free:491476kB min:760kB low:948kB high:1140kB active:4608kB inactive:132kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 3 Normal free:479044kB min:760kB low:948kB high:1140kB active:11880kB inactive:4840kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 28*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 112kB
Node 0 DMA32: 1*4kB 2*8kB 1*16kB 1*32kB 2*64kB 0*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 874*4096kB = 3587780kB
Node 0 Normal: 61*4kB 84*8kB 25*16kB 17*32kB 2*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 459*4096kB = 1884740kB
Node 1 Normal: 154*4kB 59*8kB 14*16kB 1*32kB 1*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 119*4096kB = 492032kB
Node 2 Normal: 111*4kB 63*8kB 27*16kB 11*32kB 2*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 119*4096kB = 491588kB
Node 3 Normal: 96*4kB 79*8kB 24*16kB 10*32kB 2*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 116*4096kB = 479160kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 0kB
Total swap = 0kB
Free swap:            0kB
1966080 pages of RAM
210493 reserved pages
14803 pages shared
0 pages swap cached
No available memory (MPOL_BIND): kill process 1897 (hwup) score 0 or a child

Call Trace:
 [<ffffffff80230e35>] printk_ratelimit+0x15/0x20
hwup invoked oom-killer: gfp_mask=0x40d1, order=2, oomkilladj=-17
 [<ffffffff802628d4>] out_of_memory+0x254/0x2b0

Call Trace:
 [<ffffffff80230e35>] printk_ratelimit+0x15/0x20
 [<ffffffff802628d4>] out_of_memory+0x254/0x2b0
 [<ffffffff80215a92>] smp_call_function+0x42/0x70
 [<ffffffff802647fc>] __alloc_pages+0x27c/0x3d0
 [<ffffffff80291f8c>] path_release+0x2c/0x40
 [<ffffffff8027ea8a>] alloc_pages_current+0x5a/0x90
 [<ffffffff80285014>] new_slab+0x244/0x290
 [<ffffffff80286154>] kmem_cache_alloc+0x134/0x1a0
 [<ffffffff80286551>] __kmalloc+0x21/0x30
 [<ffffffff802a1ade>] expand_files+0xce/0x300
 [<ffffffff8022d881>] dup_fd+0x2a1/0x2f0
 [<ffffffff8022d924>] copy_files+0x54/0x80
 [<ffffffff80247702>] ktime_get_ts+0x52/0x60
 [<ffffffff8022e307>] copy_process+0x547/0x1420
 [<ffffffff8022f24d>] do_fork+0x6d/0x1c0
 [<ffffffff80239612>] recalc_sigpending+0x12/0x20
 [<ffffffff8023968a>] sigprocmask+0x6a/0xf0
 [<ffffffff80209aae>] system_call+0x7e/0x83
 [<ffffffff80207f63>] sys_clone+0x23/0x30
 [<ffffffff80209dc7>] ptregscall_common+0x67/0xb0

Mem-info:
Node 0 DMA per-cpu:
CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    2: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    3: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 0 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 1 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 2 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 3 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   1   Cold: hi:   62, btch:  15 usd:   0
Active:3861 inactive:1312 dirty:0 writeback:0 unstable:0
 free:1736204 slab:2538 mapped:674 pagetables:335 bounce:0
Node 0 DMA free:348kB min:12kB low:12kB high:16kB active:0kB inactive:0kB present:9636kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 3440 5424 5424
Node 0 DMA32 free:3587656kB min:5292kB low:6612kB high:7936kB active:0kB inactive:0kB present:3523008kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1984 1984
Node 0 Normal free:1886200kB min:3052kB low:3812kB high:4576kB active:2080kB inactive:228kB present:2031616kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 1 Normal free:495236kB min:760kB low:948kB high:1140kB active:896kB inactive:108kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 2 Normal free:493600kB min:760kB low:948kB high:1140kB active:2832kB inactive:56kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 3 Normal free:481776kB min:760kB low:948kB high:1140kB active:9636kB inactive:4856kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 38*4kB 24*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 360kB
Node 0 DMA32: 1*4kB 2*8kB 1*16kB 1*32kB 2*64kB 0*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 874*4096kB = 3587780kB
Node 0 Normal: 123*4kB 86*8kB 71*16kB 28*32kB 7*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 459*4096kB = 1886412kB
Node 1 Normal: 65*4kB 84*8kB 68*16kB 38*32kB 12*64kB 5*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 119*4096kB = 495396kB
Node 2 Normal: 172*4kB 130*8kB 70*16kB 20*32kB 4*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 119*4096kB = 493728kB
Node 3 Normal: 36*4kB 96*8kB 69*16kB 29*32kB 14*64kB 7*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 116*4096kB = 481920kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 0kB
Total swap = 0kB
Free swap:            0kB
1966080 pages of RAM
210493 reserved pages
7736 pages shared
0 pages swap cached
No available memory (MPOL_BIND): kill process 1981 (hwup) score 0 or a child
hwup invoked oom-killer: gfp_mask=0x40d1, order=2, oomkilladj=-17

Call Trace:
 [<ffffffff80230e35>] printk_ratelimit+0x15/0x20
 [<ffffffff802628d4>] out_of_memory+0x254/0x2b0
 [<ffffffff80215a92>] smp_call_function+0x42/0x70
 [<ffffffff802647fc>] __alloc_pages+0x27c/0x3d0
 [<ffffffff80291f8c>] path_release+0x2c/0x40
 [<ffffffff80215a92>] smp_call_function+0x42/0x70
 [<ffffffff8027ea8a>] alloc_pages_current+0x5a/0x90
 [<ffffffff802647fc>] __alloc_pages+0x27c/0x3d0
 [<ffffffff80285014>] new_slab+0x244/0x290
 [<ffffffff8026b7ea>] __inc_zone_page_state+0x2a/0x30
 [<ffffffff80286154>] kmem_cache_alloc+0x134/0x1a0
 [<ffffffff8027ea8a>] alloc_pages_current+0x5a/0x90
 [<ffffffff80286551>] __kmalloc+0x21/0x30
 [<ffffffff80285014>] new_slab+0x244/0x290
 [<ffffffff802a1ade>] expand_files+0xce/0x300
 [<ffffffff80286154>] kmem_cache_alloc+0x134/0x1a0
 [<ffffffff8022d881>] dup_fd+0x2a1/0x2f0
 [<ffffffff8022de80>] copy_process+0xc0/0x1420
 [<ffffffff8022d924>] copy_files+0x54/0x80
 [<ffffffff80241714>] alloc_pid+0x1d4/0x320
 [<ffffffff80247702>] ktime_get_ts+0x52/0x60
 [<ffffffff8022f24d>] do_fork+0x6d/0x1c0
 [<ffffffff8022e307>] copy_process+0x547/0x1420
 [<ffffffff80239612>] recalc_sigpending+0x12/0x20
 [<ffffffff8022f24d>] do_fork+0x6d/0x1c0
 [<ffffffff8023968a>] sigprocmask+0x6a/0xf0
 [<ffffffff80239612>] recalc_sigpending+0x12/0x20
 [<ffffffff80209aae>] system_call+0x7e/0x83
 [<ffffffff8023968a>] sigprocmask+0x6a/0xf0
 [<ffffffff80207f63>] sys_clone+0x23/0x30
 [<ffffffff80209aae>] system_call+0x7e/0x83
 [<ffffffff80209dc7>] ptregscall_common+0x67/0xb0
 [<ffffffff80207f63>] sys_clone+0x23/0x30

Mem-info:
 [<ffffffff80209dc7>] ptregscall_common+0x67/0xb0
Node 0 DMA per-cpu:
CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    2: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    3: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0

Mem-info:
Node 0 DMA per-cpu:
CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    2: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    3: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 0 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 0 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 1 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:  22   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 2 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 1 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 3 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Active:3853 inactive:1316 dirty:0 writeback:0 unstable:0
 free:1736289 slab:2539 mapped:652 pagetables:313 bounce:0
Node 0 DMA free:344kB min:12kB low:12kB high:16kB active:0kB inactive:0kB present:9636kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 3440 5424 5424
Node 0 DMA32 free:3587656kB min:5292kB low:6612kB high:7936kB active:0kB inactive:0kB present:3523008kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1984 1984
Node 0 Normal free:1886304kB min:3052kB low:3812kB high:4576kB active:2128kB inactive:244kB present:2031616kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 1 Normal free:495136kB min:760kB low:948kB high:1140kB active:1100kB inactive:108kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 2 Normal free:493868kB min:760kB low:948kB high:1140kB active:2616kB inactive:56kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 3 Normal free:481848kB min:760kB low:948kB high:1140kB active:9568kB inactive:4856kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 37*4kB 25*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 364kB
Node 0 DMA32: 1*4kB 2*8kB 1*16kB 1*32kB 2*64kB 0*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 874*4096kB = 3587780kB
Node 0 Normal: 123*4kB 86*8kB 71*16kB 28*32kB 7*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 459*4096kB = 1886412kB
Node 1 Normal: CPU    1: Hot: hi:  186, btch:  31 usd:  21   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 2 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
1*4kB 80*8kB 69*16kB 38*32kB 12*64kB 5*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 119*4096kB = 495124kB
Node 2 Normal: CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
183*4kB 138*8kB 74*16kB 23*32kB 4*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 119*4096kB = 493996kB
Node 3 Normal: Node 3 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
35*4kB 96*8kB 69*16kB 29*32kB 14*64kB 7*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 116*4096kB = 481916kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 0kB
Total swap = 0kB
Free swap:            0kB
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Active:3853 inactive:1316 dirty:0 writeback:0 unstable:0
 free:1736289 slab:2539 mapped:652 pagetables:313 bounce:0
Node 0 DMA free:344kB min:12kB low:12kB high:16kB active:0kB inactive:0kB present:9636kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 3440 5424 5424
Node 0 DMA32 free:3587656kB min:5292kB low:6612kB high:7936kB active:0kB inactive:0kB present:3523008kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1984 1984
Node 0 Normal free:1886304kB min:3052kB low:3812kB high:4576kB active:2128kB inactive:244kB present:2031616kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 1 Normal free:495136kB min:760kB low:948kB high:1140kB active:1100kB inactive:108kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 2 Normal free:493868kB min:760kB low:948kB high:1140kB active:2616kB inactive:56kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 3 Normal free:481848kB min:760kB low:948kB high:1140kB active:9568kB inactive:4856kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 37*4kB 25*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 364kB
Node 0 DMA32: 1*4kB 2*8kB 1*16kB 1*32kB 2*64kB 0*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 874*4096kB = 3587780kB
Node 0 Normal: 123*4kB 86*8kB 71*16kB 28*32kB 7*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 459*4096kB = 1886412kB
Node 1 Normal: 1*4kB 80*8kB 69*16kB 38*32kB 12*64kB 5*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 119*4096kB = 495124kB
Node 2 Normal: 183*4kB 138*8kB 74*16kB 23*32kB 4*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 119*4096kB = 493996kB
Node 3 Normal: 35*4kB 96*8kB 69*16kB 29*32kB 14*64kB 7*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 116*4096kB = 481916kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 0kB
Total swap = 0kB
Free swap:            0kB
1966080 pages of RAM
210493 reserved pages
7735 pages shared
0 pages swap cached
No available memory (MPOL_BIND): kill process 1923 (hwup) score 0 or a child
1966080 pages of RAM
210493 reserved pages
7735 pages shared
0 pages swap cached
No available memory (MPOL_BIND): kill process 1981 (hwup) score 0 or a child
startpar: page allocation failure. order:4, mode:0x40d1

Call Trace:
 [<ffffffff80230e35>] printk_ratelimit+0x15/0x20
 [<ffffffff80264857>] __alloc_pages+0x2d7/0x3d0
 [<ffffffff80228bf6>] nr_context_switches+0x46/0x60
 [<ffffffff8027ea8a>] alloc_pages_current+0x5a/0x90
ERROR: could not read /proc/stat: Cannot allocate memory
 [<ffffffff80285014>] new_slab+0x244/0x290
 [<ffffffff80286154>] kmem_cache_alloc+0x134/0x1a0
 [<ffffffff80286551>] __kmalloc+0x21/0x30
 [<ffffffff802a7fd8>] seq_read+0xc8/0x2e0
 [<ffffffff802ccef1>] proc_reg_read+0x81/0xe0
 [<ffffffff8028aa81>] vfs_read+0xe1/0x190
 [<ffffffff8028af30>] sys_read+0x50/0x90
 [<ffffffff80209aae>] system_call+0x7e/0x83

Mem-info:
Node 0 DMA per-cpu:
CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    2: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    3: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 0 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:  31   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd: 162   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd: 153   Cold: hi:   62, btch:  15 usd:   0
Node 1 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:  23   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:  48   Cold: hi:   62, btch:  15 usd:   0
Node 2 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 3 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:  25   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:  35   Cold: hi:   62, btch:  15 usd:   7
Active:3241 inactive:1376 dirty:0 writeback:0 unstable:0
 free:1736624 slab:2506 mapped:713 pagetables:251 bounce:0
Node 0 DMA free:540kB min:12kB low:12kB high:16kB active:0kB inactive:0kB present:9636kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 3440 5424 5424
Node 0 DMA32 free:3587780kB min:5292kB low:6612kB high:7936kB active:0kB inactive:0kB present:3523008kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1984 1984
Node 0 Normal free:1886096kB min:3052kB low:3812kB high:4576kB active:836kB inactive:220kB present:2031616kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 1 Normal free:495200kB min:760kB low:948kB high:1140kB active:1224kB inactive:104kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 2 Normal free:495544kB min:760kB low:948kB high:1140kB active:1368kB inactive:96kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 3 Normal free:481336kB min:760kB low:948kB high:1140kB active:9536kB inactive:5084kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 1*4kB 1*8kB 8*16kB 9*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 492kB
Node 0 DMA32: 1*4kB 2*8kB 1*16kB 1*32kB 2*64kB 0*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 874*4096kB = 3587780kB
Node 0 Normal: 1*4kB 0*8kB 44*16kB 46*32kB 11*64kB 4*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 459*4096kB = 1886276kB
Node 1 Normal: 1*4kB 52*8kB 68*16kB 37*32kB 12*64kB 6*128kB 2*256kB 0*512kB 1*1024kB 1*2048kB 119*4096kB = 495236kB
Node 2 Normal: 159*4kB 97*8kB 76*16kB 41*32kB 19*64kB 6*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 119*4096kB = 495652kB
Node 3 Normal: 72*4kB 83*8kB 39*16kB 24*32kB 12*64kB 7*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 116*4096kB = 481448kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 0kB
Total swap = 0kB
Free swap:            0kB
1966080 pages of RAM
210493 reserved pages
6669 pages shared
0 pages swap cached
startpar invoked oom-killer: gfp_mask=0x40d1, order=4, oomkilladj=0

Call Trace:
 [<ffffffff80230e35>] printk_ratelimit+0x15/0x20
 [<ffffffff802628d4>] out_of_memory+0x254/0x2b0
 [<ffffffff80215a92>] smp_call_function+0x42/0x70
 [<ffffffff802647fc>] __alloc_pages+0x27c/0x3d0
 [<ffffffff80228bf6>] nr_context_switches+0x46/0x60
 [<ffffffff8027ea8a>] alloc_pages_current+0x5a/0x90
 [<ffffffff80285014>] new_slab+0x244/0x290
 [<ffffffff80286154>] kmem_cache_alloc+0x134/0x1a0
 [<ffffffff80286551>] __kmalloc+0x21/0x30
 [<ffffffff802a7fd8>] seq_read+0xc8/0x2e0
 [<ffffffff802ccef1>] proc_reg_read+0x81/0xe0
 [<ffffffff8028aa81>] vfs_read+0xe1/0x190
 [<ffffffff8028af30>] sys_read+0x50/0x90
 [<ffffffff80209aae>] system_call+0x7e/0x83

Mem-info:
Node 0 DMA per-cpu:
CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    2: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    3: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 0 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 1 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 2 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 3 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Active:3168 inactive:1583 dirty:0 writeback:0 unstable:0
 free:1736890 slab:2589 mapped:704 pagetables:181 bounce:0
Node 0 DMA free:164kB min:12kB low:12kB high:16kB active:0kB inactive:0kB present:9636kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 3440 5424 5424
Node 0 DMA32 free:3587780kB min:5292kB low:6612kB high:7936kB active:0kB inactive:0kB present:3523008kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1984 1984
Node 0 Normal free:1887696kB min:3052kB low:3812kB high:4576kB active:716kB inactive:288kB present:2031616kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 1 Normal free:494092kB min:760kB low:948kB high:1140kB active:1628kB inactive:1012kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 2 Normal free:496156kB min:760kB low:948kB high:1140kB active:724kB inactive:104kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 3 Normal free:481672kB min:760kB low:948kB high:1140kB active:9604kB inactive:4928kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 6*4kB 10*8kB 2*16kB 2*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 200kB

telnet> send brk
Node 0 DMA32: 1*4kB 2*8kB 1*16kB 1*32kB 2*64kB 0*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 874*4096kB = 3587780kB
Node 0 Normal: 169*4kB 79*8kB 58*16kB 47*32kB 13*64kB 4*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 459*4096kB = 1887964kB
Node 1 Normal: 158*4kB 60*8kB 17*16kB 12*32kB 10*64kB 6*128kB 2*256kB 0*512kB 1*1024kB 1*2048kB 119*4096kB = 494184kB
Node 2 Normal: 74*4kB 58*8kB 50*16kB 43*32kB 24*64kB 10*128kB 4*256kB 0*512kB 0*1024kB 1*2048kB 119*4096kB = 496248kB
Node 3 Normal: 150*4kB 83*8kB 39*16kB 24*32kB 12*64kB 7*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 116*4096kB = 481760kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 0kB
Total swap = 0kB
Free swap:            0kB
1966080 pages of RAM
210493 reserved pages
5195 pages shared
0 pages swap cached
No available memory (MPOL_BIND): kill process 972 (startpar) score 0 or a child
Killed process 974 (boot.rootfsck)
startpar: page allocation failure. order:4, mode:0x40d1
Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077

Call Trace:
 [<ffffffff80230e35>] printk_ratelimit+0x15/0x20
 [<ffffffff80264857>] __alloc_pages+0x2d7/0x3d0
 [<ffffffff80228bf6>] nr_context_switches+0x46/0x60
 [<ffffffff8027ea8a>] alloc_pages_current+0x5a/0x90
 [<ffffffff80285014>] new_slab+0x244/0x290
 [<ffffffff80286154>] kmem_cache_alloc+0x134/0x1a0
 [<ffffffff80286551>] __kmalloc+0x21/0x30
 [<ffffffff802a7fd8>] seq_read+0xc8/0x2e0
 [<ffffffff802ccef1>] proc_reg_read+0x81/0xe0
 [<ffffffff8028aa81>] vfs_read+0xe1/0x190
 [<ffffffff8028af30>] sys_read+0x50/0x90
 [<ffffffff80209aae>] system_call+0x7e/0x83
ERROR: could not read /proc/stat: Cannot allocate memory
^[7^[[?25l^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h

Mem-info:
Node 0 DMA per-cpu:
CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    2: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    3: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 0 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd: 183   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:  39   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:  34   Cold: hi:   62, btch:  15 usd:   0
Node 1 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:  39   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   4   Cold: hi:   62, btch:  15 usd:   0
Node 2 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 3 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd: 167   Cold: hi:   62, btch:  15 usd:   8
Active:2171 inactive:1644 dirty:0 writeback:0 unstable:0
 free:1737676 slab:2597 mapped:398 pagetables:94 bounce:0
Node 0 DMA free:196kB min:12kB low:12kB high:16kB active:0kB inactive:0kB present:9636kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 3440 5424 5424
Node 0 DMA32 free:3587780kB min:5292kB low:6612kB high:7936kB active:0kB inactive:0kB present:3523008kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1984 1984
Node 0 Normal free:1887300kB min:3052kB low:3812kB high:4576kB active:368kB inactive:284kB present:2031616kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 1 Normal free:495500kB min:760kB low:948kB high:1140kB active:332kB inactive:1000kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 2 Normal free:496640kB min:760kB low:948kB high:1140kB active:260kB inactive:104kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 3 Normal free:483288kB min:760kB low:948kB high:1140kB active:7724kB inactive:5188kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 36*4kB 7*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 216kB
Node 0 DMA32: 1*4kB 2*8kB 1*16kB 1*32kB 2*64kB 0*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 874*4096kB = 3587780kB
Node 0 Normal: 94*4kB 72*8kB 54*16kB 48*32kB 13*64kB 4*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 459*4096kB = 1887576kB
Node 1 Normal: 52*4kB 83*8kB 61*16kB 34*32kB 12*64kB 6*128kB 2*256kB 0*512kB 1*1024kB 1*2048kB 119*4096kB = 495480kB
Node 2 Normal: 43*4kB 42*8kB 36*16kB 34*32kB 26*64kB 11*128kB 8*256kB 0*512kB 0*1024kB 1*2048kB 119*4096kB = 496764kB
Node 3 Normal: 214*4kB 143*8kB 73*16kB 37*32kB 13*64kB 8*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 116*4096kB = 483392kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 0kB
Total swap = 0kB
Free swap:            0kB
1966080 pages of RAM
210493 reserved pages
1421 pages shared
0 pages swap cached
startpar invoked oom-killer: gfp_mask=0x40d1, order=4, oomkilladj=0

Call Trace:
 [<ffffffff80230e35>] printk_ratelimit+0x15/0x20
 [<ffffffff802628d4>] out_of_memory+0x254/0x2b0
 [<ffffffff80215a92>] smp_call_function+0x42/0x70
 [<ffffffff802647fc>] __alloc_pages+0x27c/0x3d0
 [<ffffffff80228bf6>] nr_context_switches+0x46/0x60
 [<ffffffff8027ea8a>] alloc_pages_current+0x5a/0x90
 [<ffffffff80285014>] new_slab+0x244/0x290
 [<ffffffff80286154>] kmem_cache_alloc+0x134/0x1a0
 [<ffffffff80286551>] __kmalloc+0x21/0x30
 [<ffffffff802a7fd8>] seq_read+0xc8/0x2e0
 [<ffffffff80273a7f>] do_mmap_pgoff+0x61f/0x8b0
 [<ffffffff802ccef1>] proc_reg_read+0x81/0xe0
 [<ffffffff8028aa81>] vfs_read+0xe1/0x190
 [<ffffffff8028af30>] sys_read+0x50/0x90
 [<ffffffff80209aae>] system_call+0x7e/0x83

Mem-info:
Node 0 DMA per-cpu:
CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    2: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    3: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 0 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 1 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 2 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 3 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Active:2171 inactive:1644 dirty:0 writeback:0 unstable:0
 free:1738133 slab:2590 mapped:398 pagetables:94 bounce:0
Node 0 DMA free:196kB min:12kB low:12kB high:16kB active:0kB inactive:0kB present:9636kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 3440 5424 5424
Node 0 DMA32 free:3587780kB min:5292kB low:6612kB high:7936kB active:0kB inactive:0kB present:3523008kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1984 1984
Node 0 Normal free:1888292kB min:3052kB low:3812kB high:4576kB active:368kB inactive:284kB present:2031616kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 1 Normal free:495652kB min:760kB low:948kB high:1140kB active:332kB inactive:1000kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 2 Normal free:496564kB min:760kB low:948kB high:1140kB active:260kB inactive:104kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 3 Normal free:484048kB min:760kB low:948kB high:1140kB active:7724kB inactive:5188kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 36*4kB 9*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 232kB
Node 0 DMA32: 1*4kB 2*8kB 1*16kB 1*32kB 2*64kB 0*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 874*4096kB = 3587780kB
Node 0 Normal: 88*4kB 88*8kB 75*16kB 51*32kB 14*64kB 7*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 459*4096kB = 1888560kB
Node 1 Normal: 71*4kB 85*8kB 59*16kB 31*32kB 17*64kB 6*128kB 2*256kB 0*512kB 1*1024kB 1*2048kB 119*4096kB = 495764kB
Node 2 Normal: 13*4kB 41*8kB 36*16kB 34*32kB 26*64kB 11*128kB 8*256kB 0*512kB 0*1024kB 1*2048kB 119*4096kB = 496636kB
Node 3 Normal: 210*4kB 145*8kB 89*16kB 42*32kB 16*64kB 7*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 116*4096kB = 484128kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 0kB
Total swap = 0kB
Free swap:            0kB
1966080 pages of RAM
210493 reserved pages
1515 pages shared
0 pages swap cached
No available memory (MPOL_BIND): kill process 2335 (startpar) score 0 or a child
Killed process 2335 (startpar)
startpar: page allocation failure. order:4, mode:0x40d1

Call Trace:
 [<ffffffff80230e35>] printk_ratelimit+0x15/0x20
 [<ffffffff80264857>] __alloc_pages+0x2d7/0x3d0
 [<ffffffff80228bf6>] nr_context_switches+0x46/0x60
 [<ffffffff8027ea8a>] alloc_pages_current+0x5a/0x90
 [<ffffffff80285014>] new_slab+0x244/0x290
 [<ffffffff80286154>] kmem_cache_alloc+0x134/0x1a0
 [<ffffffff80286551>] __kmalloc+0x21/0x30
 [<ffffffff802a7fd8>] seq_read+0xc8/0x2e0
 [<ffffffff80273a7f>] do_mmap_pgoff+0x61f/0x8b0
 [<ffffffff802ccef1>] proc_reg_read+0x81/0xe0
 [<ffffffff8028aa81>] vfs_read+0xe1/0x190
 [<ffffffff8028af30>] sys_read+0x50/0x90
 [<ffffffff80209aae>] system_call+0x7e/0x83

Mem-info:
Node 0 DMA per-cpu:
CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    2: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    3: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 0 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 1 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 2 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 3 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Active:2171 inactive:1644 dirty:0 writeback:0 unstable:0
 free:1738133 slab:2590 mapped:398 pagetables:94 bounce:0
Node 0 DMA free:196kB min:12kB low:12kB high:16kB active:0kB inactive:0kB present:9636kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 3440 5424 5424
Node 0 DMA32 free:3587780kB min:5292kB low:6612kB high:7936kB active:0kB inactive:0kB present:3523008kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1984 1984
Node 0 Normal free:1888292kB min:3052kB low:3812kB high:4576kB active:368kB inactive:284kB present:2031616kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 1 Normal free:495652kB min:760kB low:948kB high:1140kB active:332kB inactive:1000kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 2 Normal free:496564kB min:760kB low:948kB high:1140kB active:260kB inactive:104kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 3 Normal free:484048kB min:760kB low:948kB high:1140kB active:7724kB inactive:5188kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 36*4kB 9*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 232kB
Node 0 DMA32: 1*4kB 2*8kB 1*16kB 1*32kB 2*64kB 0*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 874*4096kB = 3587780kB
Node 0 Normal: 88*4kB 88*8kB 75*16kB 51*32kB 14*64kB 7*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 459*4096kB = 1888560kB
Node 1 Normal: 71*4kB 85*8kB 59*16kB 31*32kB 17*64kB 6*128kB 2*256kB 0*512kB 1*1024kB 1*2048kB 119*4096kB = 495764kB
Node 2 Normal: 13*4kB 41*8kB 36*16kB 34*32kB 26*64kB 11*128kB 8*256kB 0*512kB 0*1024kB 1*2048kB 119*4096kB = 496636kB
Node 3 Normal: 210*4kB 145*8kB 89*16kB 42*32kB 16*64kB 7*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 116*4096kB = 484128kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 0kB
Total swap = 0kB
Free swap:            0kB
1966080 pages of RAM
210493 reserved pages
1515 pages shared
0 pages swap cached
Activating swap-devices in /etc/fstab...
^[7^[[?25l^[[1A^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25hshowconsole: Warning: the ioctl TIOCGDEV is not known by the kernel
/bin/mknod -m600 /dev/shm/root b 3 2
Checking root file system...
fsck 1.40-WIP (02-Oct-2006)
[/sbin/fsck.reiserfs (1) -- /dev/shm/root] fsck.reiserfs -a /dev/shm/root 
Reiserfs super block in block 16 on 0x302 of format 3.6 with standard journal
Blocks (total/free): 19842352/8198608 by 4096 bytes
Filesystem is clean
Filesystem seems mounted read-only. Skipping journal replay.
Checking internal tree..finished
Activating device mapper...
device-mapper: ioctl: 4.11.0-ioctl (2006-10-12) initialised: dm-devel@redhat.com
^[7^[[?25l^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h
Loading required kernel modules
^[7^[[?25l^[[1A^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25hStarting MD Raid 
telnet> q\b \b
^[7^[[?25l^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h
Waiting for udev to settle...
Scanning for LVM volume groups...
  Locking type 1 initialisation failed.
Activating LVM volume groups...
  Locking type 1 initialisation failed.
^[7^[[?25l^[[80C^[[10D^[[1;33mskipped^[[m\x0f^[8^[[?25h
showconsole: Warning: the ioctl TIOCGDEV is not known by the kernel
Checking file systems...
fsck 1.40-WIP (02-Oct-2006)
Checking all file systems.
^[7^[[?25l^[[1A^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25hrm: cannot remove `/success': Read-only file system
^[7^[[?25l^[[80C^[[10D^[[1;31mfailed^[[m\x0f^[8^[[?25h
Mounting local file systems...
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
udev on /dev type tmpfs (rw)
devpts on /dev/pts type devpts (rw,mode=0620,gid=5)
nothing was mounted
^[7^[[?25l^[[1A^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25hSetting up the hardware clock^[7^[[?25l^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h
FATAL: Module apparmor not found.
Loading AppArmor module ^[[80C^[[10D^[[1;31mfailed^[[m\x0f
- could not start AppArmor^[[80C^[[10D^[[1;31mfailed^[[m\x0f
rm: cannot remove `/tmp/ssh-vbEyrP3915/agent.3915': Read-only file system
/etc/init.d/boot.cleanup: line 48: /var/run/utmp: Read-only file system
Setting up hostname 'elm3b29'^[7^[startpar invoked oom-killer: gfp_mask=0x40d1, order=4, oomkilladj=0
[?25l^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h
Setting up loopback interface     lo        
    lo        IP address: 127.0.0.1/8   
^[7^[[?25l^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h
mv: cannot move `/v
ar/log/boot.msg' to `/var/log/boot.omsg': Read-only file system
Creating /var/log/boot.msg
Can't open or create /var/run/klogd.pid.
Call Trace:
 [<ffffffff80230e35>] printk_ratelimit+0x15/0x20
 [<ffffffff802628d4>] out_of_memory+0x254/0x2b0
 [<ffffffff80215a92>] smp_call_function+0x42/0x70
 [<ffffffff802647fc>] __alloc_pages+0x27c/0x3d0
 [<ffffffff80228bf6>] nr_context_switches+0x46/0x60
 [<ffffffff8027ea8a>] alloc_pages_current+0x5a/0x90
 [<ffffffff80285014>] new_slab+0x244/0x290
 [<ffffffff80286154>] kmem_cache_alloc+0x134/0x1a0
 [<ffffffff80286551>] __kmalloc+0x21/0x30
 [<ffffffff802a7fd8>] seq_read+0xc8/0x2e0
 [<ffffffff80273a7f>] do_mmap_pgoff+0x61f/0x8b0
 [<ffffffff802ccef1>] proc_reg_read+0x81/0xe0
 [<ffffffff8028aa81>] vfs_read+0xe1/0x190
 [<ffffffff8028af30>] sys_read+0x50/0x90
 [<ffffffff80209aae>] system_call+0x7e/0x83

Mem-info:
Node 0 DMA per-cpu:
CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    2: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    3: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 0 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:  31   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 1 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 2 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 3 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Active:1271 inactive:451 dirty:0 writeback:0 unstable:0
 free:1740655 slab:2235 mapped:391 pagetables:105 bounce:0
Node 0 DMA free:1608kB min:12kB low:12kB high:16kB active:0kB inactive:0kB present:9636kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 3440 5424 5424
Node 0 DMA32 free:3587780kB min:5292kB low:6612kB high:7936kB active:0kB inactive:0kB present:3523008kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1984 1984
Node 0 Normal free:1888632kB min:3052kB low:3812kB high:4576kB active:644kB inactive:100kB present:2031616kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 1 Normal free:496420kB min:760kB low:948kB high:1140kB active:568kB inactive:36kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 2 Normal free:496888kB min:760kB low:948kB high:1140kB active:76kB inactive:44kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 3 Normal free:491292kB min:760kB low:948kB high:1140kB active:3796kB inactive:1624kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 18*4kB 90*8kB 34*16kB 7*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1624kB
Node 0 DMA32: 1*4kB 2*8kB 1*16kB 1*32kB 2*64kB 0*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 874*4096kB = 3587780kB
Node 0 Normal: 59*4kB 84*8kB 62*16kB 38*32kB 22*64kB 10*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 459*4096kB = 1888940kB
Node 1 Normal: 73*4kB 56*8kB 46*16kB 30*32kB 21*64kB 8*128kB 5*256kB 0*512kB 1*1024kB 1*2048kB 119*4096kB = 496580kB
Node 2 Normal: 41*4kB 26*8kB 25*16kB 19*32kB 16*64kB 8*128kB 4*256kB 2*512kB 2*1024kB 1*2048kB 119*4096kB = 496996kB
Node 3 Normal: 240*4kB 154*8kB 101*16kB 61*32kB 37*64kB 18*128kB 5*256kB 1*512kB 0*1024kB 2*2048kB 116*4096kB = 491456kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 0kB
Total swap = 0kB
Free swap:            0kB
1966080 pages of RAM
210493 reserved pages
1396 pages shared
^[7^[[?25l^[[1A^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h0 pages swap cached
showconsole: Warning: the ioctl TIOCGDEV is not known by the kernel
No available memory (MPOL_BIND): kill process 2535 (startpar) score 0 or a child
Killed process 2535 (startpar)
Activating remaining swap-devices in /etc/fstab...
Adding 1048784k swap on /dev/hda1.  Priority:-1 extents:1 across:1048784k
^[7^[[?25l^[[1A^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25hSetting current sysctl status from /etc/sysctl.conf
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.conf.all.rp_filter = 1
^[7^[[?25l^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h
Enabling syn flood protection^[[80C^[[10D^[[1;32mdone^[[m\x0f
Disabling IP forwarding^[7^[[?25l^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h
^[7^[[?25l^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h
System Boot Control: The system has been ^[[80C^[[10D^[[1mset up^[[m\x0f
Failed features: ^[[80C^[[33D^[[1;31mboot.rootfsck boot.cleanup^[[m\x0f
Skipped features: ^[[80C^[[17D^[[1;33mboot.cycle^[[m\x0f
System Boot Control: Running /etc/init.d/boot.local
^[7^[[?25l^[[1A^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25hINIT: Entering runlevel: 3
blogd: Warning: the ioctl TIOCGDEV is not known by the kernel
Boot logging started on /dev/ttyS0(/dev/console) at Wed Apr  4 08:52:44 2007
Master Resource Control: previous runlevel: N, switching to runlevel: ^[[80C^[[10D^[[1m3^[[m\x0f
Initializing random number generatorchmod: changing permissions of `/var/lib/misc/random-seed': Read-only file system
^[7^[[?25l^[[80C^[[10D^[[1;31mfailed^[[m\x0f^[8^[[?25h
Starting D-BUS daemon^[7^[[?25l^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h
Starting resource manager^[7^[[?25l^[[80C^[[10D^[[1;31mfailed^[[m\x0f^[8^[[?25h
FATAL: Module apparmor not found.
Loading AppArmor module ^[[80C^[[10D^[[1;31mfailed^[[m\x0f
- could not start AppArmor^[[80C^[[10D^[[1;31mfailed^[[m\x0f
Failed to start message bus: Failed to bind socket "/var/run/dbus/system_bus_socket": Read-only file system
Starting irqbalance ^[7^[[?25l^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h
input: Power Button (FF) as /class/input/input3
ACPI: Power Button (FF) [PWRF]
input: Sleep Button (FF) as /class/input/input4
ACPI: Sleep Button (FF) [SLPF]
input: Power Button (CM) as /class/input/input5
ACPI: Power Button (CM) [PWRB]
chgrp: changing group of `/opt/ibm/ewlm/ms/usr/sbin/ewlmarmd': Read-only file system
/bin/rm: cannot remove directory `/tmp/ewlmarm4': Read-only file system
chgrp: changing group of `/tmp/ewlmarm4': Read-only file system
Starting EWLM daemon
acpid: loading ACPI modules ( ac battery button fan processor thermal ) ^[7^[[?25l^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h
acpid: will skip unsupported modules
acpid: loading ACPI laptop modules ( ) ^[7^[[?25l^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h
Starting acpid acpid: can't open socket /var/run/acpid.socket: Address already in use
startproc:  exit status of parent of /sbin/acpid: 1
^[7^[[?25l^[[80C^[[10D^[[1;31mfailed^[[m\x0f^[8^[[?25h
Starting HAL daemon^[7^[[?25l^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h
Setting up network interfaces:
hald-probe-pc-f: page allocation failure. order:3, mode:0x10d1

Call Trace:
 [<ffffffff80230e35>] printk_ratelimit+0x15/0x20
 [<ffffffff80264857>] __alloc_pages+0x2d7/0x3d0
 [<ffffffff803902fa>] kobject_get+0x1a/0x30
 [<ffffffff8027ea8a>] alloc_pages_current+0x5a/0x90
 [<ffffffff80263ede>] __get_free_pages+0xe/0x40
 [<ffffffff88029d63>] :floppy:dma_mem_alloc+0x23/0x30
 [<ffffffff8802ba86>] :floppy:floppy_open+0x2f6/0x410
 [<ffffffff802b45b0>] blkdev_open+0x0/0x80
 [<ffffffff802b40fd>] do_open+0xad/0x320
 [<ffffffff802b45b0>] blkdev_open+0x0/0x80
 [<ffffffff802b45ee>] blkdev_open+0x3e/0x80
 [<ffffffff802886e8>] __dentry_open+0xd8/0x200
 [<ffffffff802888cc>] nameidata_to_filp+0x3c/0x50
 [<ffffffff8028892a>] do_filp_open+0x4a/0x50
 [<ffffffff80288576>] get_unused_fd+0x76/0x110
 [<ffffffff8028898a>] do_sys_open+0x5a/0xf0
 [<ffffffff80288a4b>] sys_open+0x1b/0x20
 [<ffffffff80209aae>] system_call+0x7e/0x83

Mem-info:
Node 0 DMA per-cpu:
CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    2: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    3: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 0 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 1 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 2 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 3 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Active:2522 inactive:1367 dirty:0 writeback:0 unstable:0
 free:1738153 slab:2384 mapped:816 pagetables:116 bounce:0
Node 0 DMA free:948kB min:12kB low:12kB high:16kB active:0kB inactive:0kB present:9636kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 3440 5424 5424
Node 0 DMA32 free:3587780kB min:5292kB low:6612kB high:7936kB active:0kB inactive:0kB present:3523008kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1984 1984
Node 0 Normal free:1888984kB min:3052kB low:3812kB high:4576kB active:620kB inactive:116kB present:2031616kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 1 Normal free:495612kB min:760kB low:948kB high:1140kB active:1164kB inactive:164kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 2 Normal free:495536kB min:760kB low:948kB high:1140kB active:1228kB inactive:40kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 3 Normal free:483752kB min:760kB low:948kB high:1140kB active:7076kB inactive:5148kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 4*4kB 57*8kB 29*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 968kB
Node 0 DMA32: 1*4kB 2*8kB 1*16kB 1*32kB 2*64kB 0*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 874*4096kB = 3587780kB
Node 0 Normal: 129*4kB 97*8kB 17*16kB 34*32kB 18*64kB 9*128kB 4*256kB 2*512kB 0*1024kB 1*2048kB 459*4096kB = 1889116kB
Node 1 Normal: 100*4kB 84*8kB 21*16kB 17*32kB 11*64kB 10*128kB 5*256kB 0*512kB 1*1024kB 1*2048kB 119*4096kB = 495712kB
Node 2 Normal: 141*4kB 93*8kB 32*16kB 11*32kB 2*64kB 1*128kB 3*256kB 2*512kB 2*1024kB 1*2048kB 119*4096kB = 495740kB
Node 3 Normal: 191*4kB 167*8kB 77*16kB 38*32kB 27*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 116*4096kB = 483844kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 1048784kB
Total swap = 1048784kB
Free swap:       1048784kB
1966080 pages of RAM
210493 reserved pages
2941 pages shared
0 pages swap cached
    lo        
    lo        IP address: 127.0.0.1/8   
^[7^[[?25l^[[1A^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h    eth0      device: Broadcom Corporation NetXtreme BCM5701 Gigabit Ethernet (rev 15)
    eth0      configuration: eth-id-00:04:76:f0:f9:aa
ip invoked oom-killer: gfp_mask=0x40d1, order=4, oomkilladj=0

Call Trace:
 [<ffffffff80230e35>] printk_ratelimit+0x15/0x20
 [<ffffffff802628d4>] out_of_memory+0x254/0x2b0
 [<ffffffff80215a92>] smp_call_function+0x42/0x70
 [<ffffffff802647fc>] __alloc_pages+0x27c/0x3d0
 [<ffffffff8027ea8a>] alloc_pages_current+0x5a/0x90
 [<ffffffff80285014>] new_slab+0x244/0x290
 [<ffffffff80286154>] kmem_cache_alloc+0x134/0x1a0
/sbin/ifup: line 1045:  2997 Killed                  ip link set up dev $INTERFA [<ffffffff802861d9>] kmem_cache_zalloc+0x19/0x40
CE ${LLADDR:+address $LLADDR} $LINK_OPTIONS
Cannot enable interface eth0.
 [<ffffffff8041bc64>] tg3_open+0xa4/0x630
 [<ffffffff8026ca06>] __do_fault+0x196/0x3a0
 [<ffffffff8047d780>] dev_open+0x40/0x90
 [<ffffffff8047b3bd>] dev_change_flags+0x6d/0x160
 [<ffffffff804c941d>] devinet_ioctl+0x58d/0x760
 [<ffffffff8047d489>] dev_ioctl+0x249/0x320
 [<ffffffff804c992d>] inet_ioctl+0x5d/0x80
 [<ffffffff8046f596>] sock_ioctl+0x56/0x230
 [<ffffffff802979a1>] do_ioctl+0x31/0x90
 [<ffffffff80297a73>] vfs_ioctl+0x73/0x2c0
 [<ffffffff80297d51>] sys_ioctl+0x91/0xb0
 [<ffffffff80209aae>] system_call+0x7e/0x83

Mem-info:
Node 0 DMA per-cpu:
CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    2: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    3: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 0 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   1   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 1 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 2 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 3 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   1   Cold: hi:   62, btch:  15 usd:   0
Active:1734 inactive:1038 dirty:0 writeback:0 unstable:0
 free:1739355 slab:2334 mapped:591 pagetables:118 bounce:0
Node 0 DMA free:1180kB min:12kB low:12kB high:16kB active:0kB inactive:0kB present:9636kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 3440 5424 5424
Node 0 DMA32 free:3587780kB min:5292kB low:6612kB high:7936kB active:0kB inactive:0kB present:3523008kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1984 1984
Node 0 Normal free:1888220kB min:3052kB low:3812kB high:4576kB active:968kB inactive:312kB present:2031616kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 1 Normal free:496192kB min:760kB low:948kB high:1140kB active:784kB inactive:108kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 2 Normal free:496508kB min:760kB low:948kB high:1140kB active:460kB inactive:140kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 3 Normal free:487540kB min:760kB low:948kB high:1140kB active:4724kB inactive:3592kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 16*4kB 52*8kB 37*16kB 4*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1200kB
Node 0 DMA32: 1*4kB 2*8kB 1*16kB 1*32kB 2*64kB 0*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 874*4096kB = 3587780kB
Node 0 Normal: 147*4kB 111*8kB 62*16kB 27*32kB 7*64kB 3*128kB 4*256kB 2*512kB 0*1024kB 1*2048kB 459*4096kB = 1888324kB
Node 1 Normal: 106*4kB 81*8kB 42*16kB 21*32kB 13*64kB 10*128kB 5*256kB 0*512kB 1*1024kB 1*2048kB 119*4096kB = 496304kB
Node 2 Normal: 103*4kB 88*8kB 45*16kB 21*32kB 9*64kB 2*128kB 3*256kB 2*512kB 2*1024kB 1*2048kB 119*4096kB = 496652kB
Node 3 Normal: 329*4kB 190*8kB 100*16kB 53*32kB 38*64kB 7*128kB 4*256kB 0*512kB 0*1024kB 1*2048kB 116*4096kB = 487668kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 1048784kB
Total swap = 1048784kB
Free swap:       1048784kB
1966080 pages of RAM
210493 reserved pages
2219 pages shared
0 pages swap cached
No available memory (MPOL_BIND): kill process 2997 (ip) score 0 or a child
Killed process 2997 (ip)
ip: page allocation failure. order:4, mode:0x40d1

Call Trace:
 [<ffffffff80230e35>] printk_ratelimit+0x15/0x20
 [<ffffffff80264857>] __alloc_pages+0x2d7/0x3d0
 [<ffffffff8027ea8a>] alloc_pages_current+0x5a/0x90
 [<ffffffff80285014>] new_slab+0x244/0x290
 [<ffffffff80286154>] kmem_cache_alloc+0x134/0x1a0
 [<ffffffff802861d9>] kmem_cache_zalloc+0x19/0x40
 [<ffffffff8041bc64>] tg3_open+0xa4/0x630
 [<ffffffff8026ca06>] __do_fault+0x196/0x3a0
 [<ffffffff8047d780>] dev_open+0x40/0x90
 [<ffffffff8047b3bd>] dev_change_flags+0x6d/0x160
 [<ffffffff804c941d>] devinet_ioctl+0x58d/0x760
 [<ffffffff8047d489>] dev_ioctl+0x249/0x320
 [<ffffffff804c992d>] inet_ioctl+0x5d/0x80
 [<ffffffff8046f596>] sock_ioctl+0x56/0x230
 [<ffffffff802979a1>] do_ioctl+0x31/0x90
 [<ffffffff80297a73>] vfs_ioctl+0x73/0x2c0
 [<ffffffff80297d51>] sys_ioctl+0x91/0xb0
 [<ffffffff80209aae>] system_call+0x7e/0x83

Mem-info:
Node 0 DMA per-cpu:
CPU    0: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    1: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    2: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
CPU    3: Hot: hi:    0, btch:   1 usd:   0   Cold: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 0 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   1   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 1 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 2 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
Node 3 Normal per-cpu:
CPU    0: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    1: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    2: Hot: hi:  186, btch:  31 usd:   0   Cold: hi:   62, btch:  15 usd:   0
CPU    3: Hot: hi:  186, btch:  31 usd:   1   Cold: hi:   62, btch:  15 usd:   0
Active:1734 inactive:1038 dirty:0 writeback:0 unstable:0
 free:1739355 slab:2334 mapped:591 pagetables:118 bounce:0
Node 0 DMA free:1180kB min:12kB low:12kB high:16kB active:0kB inactive:0kB present:9636kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 3440 5424 5424
Node 0 DMA32 free:3587780kB min:5292kB low:6612kB high:7936kB active:0kB inactive:0kB present:3523008kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 1984 1984
Node 0 Normal free:1888220kB min:3052kB low:3812kB high:4576kB active:968kB inactive:312kB present:2031616kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 1 Normal free:496192kB min:760kB low:948kB high:1140kB active:784kB inactive:108kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 2 Normal free:496508kB min:760kB low:948kB high:1140kB active:460kB inactive:140kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 3 Normal free:487540kB min:760kB low:948kB high:1140kB active:4724kB inactive:3592kB present:507904kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 16*4kB 52*8kB 37*16kB 4*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1200kB
Node 0 DMA32: 1*4kB 2*8kB 1*16kB 1*32kB 2*64kB 0*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 874*4096kB = 3587780kB
Node 0 Normal: 147*4kB 111*8kB 62*16kB 27*32kB 7*64kB 3*128kB 4*256kB 2*512kB 0*1024kB 1*2048kB 459*4096kB = 1888324kB
Node 1 Normal: 106*4kB 81*8kB 42*16kB 21*32kB 13*64kB 10*128kB 5*256kB 0*512kB 1*1024kB 1*2048kB 119*4096kB = 496304kB
Node 2 Normal: 103*4kB 88*8kB 45*16kB 21*32kB 9*64kB 2*128kB 3*256kB 2*512kB 2*1024kB 1*2048kB 119*4096kB = 496652kB
Node 3 Normal: 329*4kB 190*8kB 100*16kB 53*32kB 38*64kB 7*128kB 4*256kB 0*512kB 0*1024kB 1*2048kB 116*4096kB = 487668kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 1048784kB
Total swap = 1048784kB
Free swap:       1048784kB
1966080 pages of RAM
210493 reserved pages
2219 pages shared
0 pages swap cached
interface eth0 is not up
^[7^[[?25l^[[1A^[[80C^[[10D^[[1;31mfailed^[[m\x0f^[8^[[?25hWaiting for mandatory devices:  eth-id-00:04:76:f0:f9:aa __NSC__
5 4 3 2 1 
    eth0      device: Broadcom Corporation NetXtreme BCM5701 Gigabit Ethernet (rev 15)
    eth0      configuration: eth-id-00:04:76:f0:f9:aa
    eth0      is down
^[7^[[?25l^[[1A^[[80C^[[10D^[[1;31mfailed^[[m\x0f^[8^[[?25h    eth0      interface could not be set up until now
^[7^[[?25l^[[1A^[[80C^[[10D^[[1;31mfailed^[[m\x0f^[8^[[?25hSetting up service network  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .^[7^[[?25l^[[80C^[[10D^[[1;31mfailed^[[m\x0f^[8^[[?25h
Starting syslog services^[7^[[?25l^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h
Starting slpd A FATAL Error has occured:
Could not open logfile /var/log/slpd.log
startproc:  exit status of parent of /usr/sbin/slpd: 1
^[7^[[?25l^[[80C^[[10D^[[1;31mfailed^[[m\x0f^[8^[[?25h
Starting RPC portmap daemon^[7^[[?25l^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h
Starting auditd ^[7^[[?25l^[[80C^[[10D^[[1;31mfailed^[[m\x0f^[8^[[?25h
Importing Net File System (NFS)^[[80C^[[10D^[[1munused^[[m\x0f
Checking/updating CPU microcode ^[7^[[?25l^[[80C^[[10D^[[1munused^[[m\x0f^[8^[[?25h
Starting nfsboot (sm-notify) ^[7^[[?25l^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h
Mount SMB/ CIFS File Systems ^[[80C^[[10D^[[1munused^[[m\x0f
touch: cannot touch `/var/lock/subsys/smbfs': Read-only file system
Starting Name Service Cache Daemon^[7^[[?25l^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h
Starting cupsd^[7^[[?25l^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
Starting SSH daemon^[7^[[?25l^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h
Starting ZENworks Management Daemon^[7^[[?25l^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h
Starting powersaved: ###############################################
# ACPI system but acpid not running.          #
# Start acpid first, then restart powersaved! #
###############################################^[[80C^[[10D^[[1;33mskipped^[[m\x0f
Starting mail service (Postfix)^[7^[[?25l^[[80C^[[10D^[[1;32mdone^[[m\x0f^[8^[[?25h
Starting CRON daemon/usr/sbin/cron: can't open or create /var/run/cron.pid: Read-only file system
startproc:  exit status of parent of /usr/sbin/cron: 1
^[7^[[?25l^[[80C^[[10D^[[1;31mfailed^[[m\x0f^[8^[[?25h
Master Resource Control: runlevel 3 has been ^[[80C^[[10D^[[1mreached^[[m\x0f
Failed services in runlevel 3: ^[[1;31macpid random resmgr network slpd auditd cron^[[m\x0f
Skipped services in runlevel 3: ^[[80C^[[44D^[[1;33msmbfs nfs microcode powersaved splash^[[m\x0f
blogd: no message logging because /var file system is not accessible



Welcome to SUSE Linux Enterprise Server 10 (x86_64) - Kernel 2.6.21-rc5-mm4 (ttyS0).



Welcome to SUSE Linux Enterprise Server 10 (x86_64) - Kernel 2.6.21-rc5-mm4 (console).


elm3b29 login: elm3b29 login: 
telnet> q
Connection closed.
^[]0;badari@dyn9047017100:~\a[badari@dyn9047017100 ~]$ 
^[]0;badari@dyn9047017100:~\a[badari@dyn9047017100 ~]$ 
^[]0;badari@dyn9047017100:~\a[badari@dyn9047017100 ~]$ exit

Script done on Wed 04 Apr 2007 07:50:14 AM PDT

Re: 2.6.21-rc5-mm4 (SLUB)

[Christoph Lameter]

On Wed, 4 Apr 2007, Badari Pulavarty wrote:

> On Tue, 2007-04-03 at 16:55 -0700, Christoph Lameter wrote:
> > On Tue, 3 Apr 2007, Badari Pulavarty wrote:
> > 
> > > Hmm. booted fine with slub_debug :(
> > 
> > Try to selectively disable debug options... if you got the 
> > time...
> > 
> > F.e. Try with sanity checks only
> > 
> > slub_debug=F
> 
> slub_debug=F got something. 

Ahh.... Seems that the first 4 bytes of the allocations is zapped after 
the object has been freed. Can you trap writes to the first four bytes of 
the object? This should give you the culprit.

The other thing is that the system is performing DMA allocations
for the file cache???? Then its running out of memory.

Argh.... We use  GFP DMA bitmask to check SLAB flags field:

Try this fix:



SLUB: Use correct flags to check for DMA cache

We use a GFP mask to check the SLAB flags if this is a DMA cache.

Fix this by using the correct SLAB mask and then use the SLUB_DMA
for the ORing of flags. If the system does not support DMA then
we will OR zero which will hopefully get the compiler to drop the
useless if statement as well.

Signed-off-by: Christoph Lameter <clameter@sgi.com>

Index: linux-2.6.21-rc5-mm4/mm/slub.c
===================================================================
--- linux-2.6.21-rc5-mm4.orig/mm/slub.c	2007-04-04 09:59:05.000000000 -0700
+++ linux-2.6.21-rc5-mm4/mm/slub.c	2007-04-04 10:01:14.000000000 -0700
@@ -678,8 +678,8 @@ static struct page *allocate_slab(struct
 	if (s->order)
 		flags |= __GFP_COMP;
 
-	if (s->flags & SLUB_DMA)
-		flags |= GFP_DMA;
+	if (s->flags & SLAB_CACHE_DMA)
+		flags |= SLUB_DMA;
 
 	if (node == -1)
 		page = alloc_pages(flags, s->order);

Re: 2.6.21-rc5-mm4 (SLUB)

[Badari Pulavarty]

On Wed, 2007-04-04 at 10:03 -0700, Christoph Lameter wrote:
> On Wed, 4 Apr 2007, Badari Pulavarty wrote:
> 
> > On Tue, 2007-04-03 at 16:55 -0700, Christoph Lameter wrote:
> > > On Tue, 3 Apr 2007, Badari Pulavarty wrote:
> > > 
> > > > Hmm. booted fine with slub_debug :(
> > > 
> > > Try to selectively disable debug options... if you got the 
> > > time...
> > > 
> > > F.e. Try with sanity checks only
> > > 
> > > slub_debug=F
> > 
> > slub_debug=F got something. 
> 
> Ahh.... Seems that the first 4 bytes of the allocations is zapped after 
> the object has been freed. Can you trap writes to the first four bytes of 
> the object? This should give you the culprit.
> 
> The other thing is that the system is performing DMA allocations
> for the file cache???? Then its running out of memory.
> 
> Argh.... We use  GFP DMA bitmask to check SLAB flags field:
> 
> Try this fix:
> 
> 
> 
> SLUB: Use correct flags to check for DMA cache
> 
> We use a GFP mask to check the SLAB flags if this is a DMA cache.
> 
> Fix this by using the correct SLAB mask and then use the SLUB_DMA
> for the ORing of flags. If the system does not support DMA then
> we will OR zero which will hopefully get the compiler to drop the
> useless if statement as well.
> 
> Signed-off-by: Christoph Lameter <clameter@sgi.com>
> 
> Index: linux-2.6.21-rc5-mm4/mm/slub.c
> ===================================================================
> --- linux-2.6.21-rc5-mm4.orig/mm/slub.c	2007-04-04 09:59:05.000000000 -0700
> +++ linux-2.6.21-rc5-mm4/mm/slub.c	2007-04-04 10:01:14.000000000 -0700
> @@ -678,8 +678,8 @@ static struct page *allocate_slab(struct
>  	if (s->order)
>  		flags |= __GFP_COMP;
>  
> -	if (s->flags & SLUB_DMA)
> -		flags |= GFP_DMA;
> +	if (s->flags & SLAB_CACHE_DMA)
> +		flags |= SLUB_DMA;
>  
>  	if (node == -1)
>  		page = alloc_pages(flags, s->order);
> 


Machine booted fine with slub_debug=F. Got following in the
log. I guess we need to track down who is touching after
free. Any ideas on how I can track down easily ? Is there
a way to store last allocated (function, line#) and look
around there ?

Thanks,
Badari

*** SLUB: Freepointer corrupt in kmalloc-16@0xffff810173f172a0 Slab
0xffff81017f9f8b80
    offset=672 flags=0x2000000000000c7 inuse=42
freelist=0xffff810173f172a0
  Bytes b4 0xffff810173f17290:  a0 72 f1 73 00 00 00 00 00 00 00 00 00
00 00 00 .r\uffffs............
    Object 0xffff810173f172a0:  00 00 00 00 01 81 ff ff 00 00 00 00 00
00 00 00 ......\uffff\uffff........
FreePointer 0xffff810173f172a0 -> 0xffff810100000000

Call Trace:
 [<ffffffff802840e5>] object_err+0x105/0x1b0
 [<ffffffff80284485>] check_object+0x1b5/0x1d0
 [<ffffffff80284ad4>] alloc_object_checks+0x64/0x110
 [<ffffffff8028611c>] kmem_cache_alloc+0xfc/0x1a0
 [<ffffffff802dcb27>] sysfs_create_link+0xb7/0x160
 [<ffffffff8024f431>] module_add_driver+0x41/0xd0
 [<ffffffff8040241e>] bus_add_driver+0xce/0x1d0
 [<ffffffff8040324d>] driver_register+0x5d/0x90
 [<ffffffff803a16a8>] __pci_register_driver+0x68/0xb0
 [<ffffffff806d6066>] agp_amd64_init+0x36/0xe0
 [<ffffffff806c3ad6>] gart_iommu_init+0x4c6/0x560
 [<ffffffff80227a8e>] __wake_up+0x4e/0x70
 [<ffffffff80495eb0>] genl_rcv+0x0/0x70
 [<ffffffff8049432c>] netlink_kernel_create+0x14c/0x160
 [<ffffffff80495cd0>] genl_unlock+0x10/0x40
 [<ffffffff806bc8de>] pci_iommu_init+0xe/0x20
 [<ffffffff806b2a04>] kernel_init+0x154/0x330
 [<ffffffff8020a8c8>] child_rip+0xa/0x12
 [<ffffffff806b28b0>] kernel_init+0x0/0x330
 [<ffffffff8020a8be>] child_rip+0x0/0x12

Re: 2.6.21-rc5-mm4 (SLUB)

[Christoph Lameter]

On Wed, 4 Apr 2007, Badari Pulavarty wrote:

> Machine booted fine with slub_debug=F. Got following in the
> log. I guess we need to track down who is touching after
> free. Any ideas on how I can track down easily ? Is there
> a way to store last allocated (function, line#) and look
> around there ?

Yes. slub_debug=U. But user tracking may need to increase the slab 
size (depends on the padding available in the slab) to store the 
tracking information, so you may not get the same corruption.

> *** SLUB: Freepointer corrupt in kmalloc-16@0xffff810173f172a0 Slab
> 0xffff81017f9f8b80
>     offset=672 flags=0x2000000000000c7 inuse=42
> freelist=0xffff810173f172a0
>   Bytes b4 0xffff810173f17290:  a0 72 f1 73 00 00 00 00 00 00 00 00 00
> 00 00 00 .r\uffffs............
>     Object 0xffff810173f172a0:  00 00 00 00 01 81 ff ff 00 00 00 00 00
> 00 00 00 ......\uffff\uffff........
> FreePointer 0xffff810173f172a0 -> 0xffff810100000000

Same as before.

Re: 2.6.21-rc5-mm4 (SLUB)

[Christoph Lameter]

On Wed, 4 Apr 2007, Christoph Lameter wrote:

> Yes. slub_debug=U. But user tracking may need to increase the slab 
> size (depends on the padding available in the slab) to store the 
> tracking information, so you may not get the same corruption.

Hummm U is switching off merging and you may need merging to trigger the 
discovery of the overwrite.

Here is a patch to enable merging even while tracking slabs. This patch 
should not be applied to mm. In general tracking requires knowing which
slab the objects come from and merging looses that information.

Index: linux-2.6.21-rc5-mm4/mm/slub.c
===================================================================
--- linux-2.6.21-rc5-mm4.orig/mm/slub.c	2007-04-04 11:19:29.000000000 -0700
+++ linux-2.6.21-rc5-mm4/mm/slub.c	2007-04-04 11:19:35.000000000 -0700
@@ -86,7 +86,7 @@
 /*
  * Set of flags that will prevent slab merging
  */
-#define SLUB_NEVER_MERGE (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER | \
+#define SLUB_NEVER_MERGE (SLAB_RED_ZONE | SLAB_POISON | \
 		SLAB_TRACE | SLAB_DESTROY_BY_RCU)
 
 #define SLUB_MERGE_SAME (SLAB_DEBUG_FREE | SLAB_RECLAIM_ACCOUNT | \

Re: 2.6.21-rc5-mm4 (SLUB)

[Badari Pulavarty]

[-- Attachment #1: Type: text/plain, Size: 1323 bytes --]

On Wed, 2007-04-04 at 11:22 -0700, Christoph Lameter wrote:
> On Wed, 4 Apr 2007, Christoph Lameter wrote:
> 
> > Yes. slub_debug=U. But user tracking may need to increase the slab 
> > size (depends on the padding available in the slab) to store the 
> > tracking information, so you may not get the same corruption.
> 
> Hummm U is switching off merging and you may need merging to trigger the 
> discovery of the overwrite.
> 
> Here is a patch to enable merging even while tracking slabs. This patch 
> should not be applied to mm. In general tracking requires knowing which
> slab the objects come from and merging looses that information.
> 
> Index: linux-2.6.21-rc5-mm4/mm/slub.c
> ===================================================================
> --- linux-2.6.21-rc5-mm4.orig/mm/slub.c	2007-04-04 11:19:29.000000000 -0700
> +++ linux-2.6.21-rc5-mm4/mm/slub.c	2007-04-04 11:19:35.000000000 -0700
> @@ -86,7 +86,7 @@
>  /*
>   * Set of flags that will prevent slab merging
>   */
> -#define SLUB_NEVER_MERGE (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER | \
> +#define SLUB_NEVER_MERGE (SLAB_RED_ZONE | SLAB_POISON | \
>  		SLAB_TRACE | SLAB_DESTROY_BY_RCU)
>  
>  #define SLUB_MERGE_SAME (SLAB_DEBUG_FREE | SLAB_RECLAIM_ACCOUNT | \
> 

Here is the slub_debug=FU output with the above patch.

Thanks,
Badari



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: slub_debug=FU --]
[-- Type: text/plain; name="slub_debug=FU"; charset=utf-8, Size: 36122 bytes --]

Linux version 2.6.21-rc5-mm4 (root@elm3b29) (gcc version 4.1.0 (SUSE Linux)) #6 SMP Wed Apr 4 16:52:03 PDT 2007
Command line: root=/dev/hda2 vga=0x314  slub_debug=FU selinux=0   console=tty0 console=ttyS0,38400 resume=/dev/hda1 resume=/dev/hda1  splash=silent showopts
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009f000 (usable)
 BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000ca000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 00000000dfef0000 (usable)
 BIOS-e820: 00000000dfef0000 - 00000000dfeff000 (ACPI data)
 BIOS-e820: 00000000dfeff000 - 00000000dff00000 (ACPI NVS)
 BIOS-e820: 00000000dff00000 - 00000000e0000000 (usable)
 BIOS-e820: 00000000fec00000 - 00000000fec00400 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
 BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
 BIOS-e820: 0000000100000000 - 00000001e0000000 (usable)
end_pfn_map = 1966080
DMI 2.3 present.
ACPI: RSDP 000F6970, 0024 (r2 PTLTD )
ACPI: XSDT DFEFC625, 003C (r1 PTLTD  	 XSDT    6040000  LTP        0)
ACPI: FACP DFEFED02, 00F4 (r3 AMD    HAMMER    6040000 PTEC    F4240)
ACPI: DSDT DFEFC661, 262D (r1 AMD-K8  AMDACPI  6040000 MSFT  100000D)
ACPI: FACS DFEFFFC0, 0040
ACPI: SRAT DFEFEDF6, 0160 (r1 AMD    HAMMER    6040000 AMD         1)
ACPI: APIC DFEFEF56, 00AA (r1 PTLTD  	 APIC    6040000  LTP        0)
SRAT: PXM 0 -> APIC 0 -> Node 0
SRAT: PXM 1 -> APIC 1 -> Node 1
SRAT: PXM 2 -> APIC 2 -> Node 2
SRAT: PXM 3 -> APIC 3 -> Node 3
SRAT: Node 0 PXM 0 0-a0000
SRAT: Node 0 PXM 0 0-e0000000
SRAT: Node 0 PXM 0 0-180000000
SRAT: PXM 1 (100000000-1a0000000) overlaps with PXM 0 (0-180000000)
SRAT: SRAT not used.
Scanning NUMA topology in Northbridge 24
Number of nodes 4
Node 0 MemBase 0000000000000000 Limit 0000000180000000
Node 1 MemBase 0000000180000000 Limit 00000001a0000000
Node 2 MemBase 00000001a0000000 Limit 00000001c0000000
Node 3 MemBase 00000001c0000000 Limit 00000001e0000000
Using node hash shift of 29
Bootmem setup node 0 0000000000000000-0000000180000000
Bootmem setup node 1 0000000180000000-00000001a0000000
Bootmem setup node 2 00000001a0000000-00000001c0000000
Bootmem setup node 3 00000001c0000000-00000001e0000000
Zone PFN ranges:
  DMA             0 ->     4096
  DMA32        4096 ->  1048576
  Normal    1048576 ->  1966080
Movable zone start PFN for each node
early_node_map[7] active PFN ranges
    0:        0 ->      159
    0:      256 ->   917232
    0:   917248 ->   917504
    0:  1048576 ->  1572864
    1:  1572864 ->  1703936
    2:  1703936 ->  1835008
    3:  1835008 ->  1966080
ACPI: PM-Timer IO Port: 0x8008
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Processor #0 (Bootup-CPU)
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
Processor #1
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
Processor #2
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
Processor #3
ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 4, address 0xfec00000, GSI 0-23
ACPI: IOAPIC (id[0x05] address[0xfa3e0000] gsi_base[24])
IOAPIC[1]: apic_id 5, address 0xfa3e0000, GSI 24-27
ACPI: IOAPIC (id[0x06] address[0xfa3e1000] gsi_base[28])
IOAPIC[2]: apic_id 6, address 0xfa3e1000, GSI 28-31
ACPI: IOAPIC (id[0x07] address[0xfa3e2000] gsi_base[32])
IOAPIC[3]: apic_id 7, address 0xfa3e2000, GSI 32-35
ACPI: IOAPIC (id[0x08] address[0xfa3e4000] gsi_base[36])
IOAPIC[4]: apic_id 8, address 0xfa3e4000, GSI 36-39
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
Setting APIC routing to flat
Using ACPI (MADT) for SMP configuration information
Allocating PCI resources starting at e2000000 (gap: e0000000:1ec00000)
PERCPU: Allocating 41984 bytes of per cpu data
Built 4 zonelists.  Total pages: 1771993
Kernel command line: root=/dev/hda2 vga=0x314  slub_debug=FU selinux=0   console=tty0 console=ttyS0,38400 resume=/dev/hda1 resume=/dev/hda1  splash=silent showopts
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 32768 bytes)
Extended CMOS year: 2000
time.c: Detected 1398.178 MHz processor.
Console: colour dummy device 80x25
Checking aperture...
CPU 0: aperture @ 0 size 32 MB
No AGP bridge found
Your BIOS doesn't leave a aperture memory hole
Please enable the IOMMU option in the BIOS setup
This costs you 64 MB of RAM
Mapping aperture over 65536 KB of RAM @ 4000000
Memory: 7021948k/7864320k available (3113k kernel code, 317632k reserved, 1229k data, 364k init)
SLUB V6: General Slabs=18, HW alignment=128, Processors=4, Nodes=64
Calibrating delay using timer specific routine.. 2799.04 BogoMIPS (lpj=5598086)
Security Framework v1.0.0 initialized
Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
Mount-cache hash table entries: 256
*** SLUB: Freepointer corrupt in kmalloc-32@0xffff810180033190 Slab 0xffff81019f001980
    offset=400 flags=0x6000000000000c7 inuse=5 freelist=0xffff810180033190
  Bytes b4 0xffff810180033180:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff810180033190:  91 65 28 80 ff ff ff ff 00 00 00 00 00 00 00 00 .e(.ÿÿÿÿ........
    Object 0xffff8101800331a0:  62 db fe ff 00 00 00 00 00 00 00 00 00 00 00 00 bÛþÿ............
    Object 0xffff8101800331b0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff8101800331c0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff8101800331d0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
FreePointer 0xffff810180033190 -> 0xffffffff80286591
Last alloc: 0xffff810180033230 jiffies since=4294892386 cpu=0 pid=0
    Filler 0xffff810180033210:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810180033220:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810180033230:  80 32 03 80 01 81 ff ff 00 00 00 00 00 00 00 00 .2....ÿÿ........
    Filler 0xffff810180033240:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810180033250:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810180033260:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810180033270:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810180033280:  d0 32 03 80 01 81 ff ff 00 00 00 00 00 00 00 00 Ð2....ÿÿ........

Call Trace:
 [<ffffffff802840e5>] object_err+0x105/0x1b0
 [<ffffffff80284485>] check_object+0x1b5/0x1d0
 [<ffffffff80284ad4>] alloc_object_checks+0x64/0x110
 [<ffffffff8028611c>] kmem_cache_alloc+0xfc/0x1a0
 [<ffffffff802861d9>] kmem_cache_zalloc+0x19/0x40
 [<ffffffff802db6ad>] __sysfs_new_dirent+0x1d/0x60
 [<ffffffff802db8bc>] __sysfs_make_dirent+0x2c/0xb0
 [<ffffffff802db965>] sysfs_make_dirent+0x25/0x50
 [<ffffffff802dbd64>] create_dir+0xc4/0x200
 [<ffffffff802dbecd>] sysfs_create_dir+0x2d/0x80
 [<ffffffff803902fa>] kobject_get+0x1a/0x30
 [<ffffffff803905a9>] kobject_shadow_add+0xb9/0x1e0
 [<ffffffff803906db>] kobject_add+0xb/0x10
 [<ffffffff803906f4>] kset_add+0x14/0x30
 [<ffffffff80390aa3>] subsystem_register+0x23/0x40
 [<ffffffff806ce96d>] mnt_init+0xed/0x240
 [<ffffffff806ce4d9>] vfs_caches_init+0xd9/0x180
 [<ffffffff806b27fb>] start_kernel+0x21b/0x2d0
 [<ffffffff806b2127>] _sinittext+0x127/0x170

*** SLUB: Freepointer corrupt in kmalloc-16@0xffff810173c00440 Slab 0xffff81017f9e0000
    offset=1088 flags=0x2000000000000c7 inuse=17 freelist=0xffff810173c00440
  Bytes b4 0xffff810173c00430:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff810173c00440:  1f 65 28 80 ff ff ff ff 00 00 00 00 00 00 00 00 .e(.ÿÿÿÿ........
    Object 0xffff810173c00450:  5e db fe ff 00 00 00 00 00 00 00 00 00 00 00 00 ^Ûþÿ............
    Object 0xffff810173c00460:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff810173c00470:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
FreePointer 0xffff810173c00440 -> 0xffffffff8028651f
Last alloc: 0xffff810173c004c0 jiffies since=4294892387 cpu=0 pid=0
    Filler 0xffff810173c004b0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810173c004c0:  00 05 c0 73 01 81 ff ff 00 e8 3a 80 01 81 ff ff ..Às..ÿÿ.è:...ÿÿ
    Filler 0xffff810173c004d0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810173c004e0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810173c004f0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810173c00500:  40 05 c0 73 01 81 ff ff 00 00 00 00 00 00 00 00 @.Às..ÿÿ........
    Filler 0xffff810173c00510:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810173c00520:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................

Call Trace:
 [<ffffffff802840e5>] object_err+0x105/0x1b0
 [<ffffffff80284485>] check_object+0x1b5/0x1d0
 [<ffffffff80284ad4>] alloc_object_checks+0x64/0x110
 [<ffffffff8028611c>] kmem_cache_alloc+0xfc/0x1a0
 [<ffffffff80286551>] __kmalloc+0x21/0x30
 [<ffffffff802d0f5d>] proc_symlink+0x4d/0xb0
 [<ffffffff802d3020>] execdomains_read_proc+0x0/0x60
 [<ffffffff806cf4e5>] proc_misc_init+0x75/0x210
 [<ffffffff806cf334>] proc_root_init+0x44/0xf0
 [<ffffffff806b280f>] start_kernel+0x22f/0x2d0
 [<ffffffff806b2127>] _sinittext+0x127/0x170

CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 0/0 -> Node 0
*** SLUB: Freepointer corrupt in kmalloc-32@0xffff8101802fd050 Slab 0xffff81019f017e80
    offset=80 flags=0x6000000000000c7 inuse=1 freelist=0xffff8101802fd050
  Bytes b4 0xffff8101802fd040:  00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 ................
    Object 0xffff8101802fd050:  d9 61 28 80 ff ff ff ff 00 00 00 00 00 00 00 00 Ùa(.ÿÿÿÿ........
    Object 0xffff8101802fd060:  63 db fe ff 00 00 00 00 00 00 00 00 00 00 00 00 cÛþÿ............
    Object 0xffff8101802fd070:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff8101802fd080:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff8101802fd090:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
FreePointer 0xffff8101802fd050 -> 0xffffffff802861d9
Last alloc: 0xffff8101802fd0f0 jiffies since=4294892391 cpu=0 pid=0
    Filler 0xffff8101802fd0d0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101802fd0e0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101802fd0f0:  40 d1 2f 80 01 81 ff ff 00 00 00 00 00 00 00 00 @Ñ/...ÿÿ........
    Filler 0xffff8101802fd100:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101802fd110:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101802fd120:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101802fd130:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101802fd140:  90 d1 2f 80 01 81 ff ff 00 00 00 00 00 00 00 00 .Ñ/...ÿÿ........

Call Trace:
 [<ffffffff802840e5>] object_err+0x105/0x1b0
 [<ffffffff80284485>] check_object+0x1b5/0x1d0
 [<ffffffff80284ad4>] alloc_object_checks+0x64/0x110
 [<ffffffff8028611c>] kmem_cache_alloc+0xfc/0x1a0
 [<ffffffff80286551>] __kmalloc+0x21/0x30
 [<ffffffff806bdbc0>] mtrr_bp_init+0xd0/0x1e0
 [<ffffffff806babca>] identify_cpu+0x6fa/0x7e0
 [<ffffffff806b281b>] start_kernel+0x23b/0x2d0
 [<ffffffff806b2127>] _sinittext+0x127/0x170

Freeing SMP alternatives: 36k freed
ACPI: Core revision 20070126
*** SLUB: Freepointer corrupt in kmalloc-32@0xffff8101a02fe050 Slab 0xffff8101bf017f00
    offset=80 flags=0xa000000000000c7 inuse=1 freelist=0xffff8101a02fe050
  Bytes b4 0xffff8101a02fe040:  98 14 ed 70 01 81 ff ff 00 00 00 00 00 00 00 00 ..íp..ÿÿ........
    Object 0xffff8101a02fe050:  51 65 28 80 ff ff ff ff 00 00 00 00 00 00 00 00 Qe(.ÿÿÿÿ........
    Object 0xffff8101a02fe060:  68 db fe ff 00 00 00 00 00 00 00 00 00 00 00 00 hÛþÿ............
    Object 0xffff8101a02fe070:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff8101a02fe080:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff8101a02fe090:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
FreePointer 0xffff8101a02fe050 -> 0xffffffff80286551
Last alloc: 0xffff8101a02fe0f0 jiffies since=4294892394 cpu=0 pid=0
    Filler 0xffff8101a02fe0d0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101a02fe0e0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101a02fe0f0:  40 e1 2f a0 01 81 ff ff 00 00 00 00 00 00 00 00 @á/...ÿÿ........
    Filler 0xffff8101a02fe100:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101a02fe110:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101a02fe120:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101a02fe130:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101a02fe140:  90 e1 2f a0 01 81 ff ff 00 00 00 00 00 00 00 00 .á/...ÿÿ........

Call Trace:
 [<ffffffff802840e5>] object_err+0x105/0x1b0
 [<ffffffff80284485>] check_object+0x1b5/0x1d0
 [<ffffffff80284ad4>] alloc_object_checks+0x64/0x110
 [<ffffffff8028611c>] kmem_cache_alloc+0xfc/0x1a0
 [<ffffffff80286591>] sysfs_slab_alias+0x31/0x80
 [<ffffffff80286afc>] kmem_cache_create+0x20c/0x220
 [<ffffffff803b76ea>] acpi_os_create_cache+0x1e/0x2e
 [<ffffffff803cc664>] acpi_ut_create_caches+0x46/0xae
 [<ffffffff803ce560>] acpi_ut_init_globals+0x9/0x1a1
 [<ffffffff803cda3b>] acpi_initialize_subsystem+0x3a/0x8b
 [<ffffffff806d382b>] acpi_early_init+0x5c/0x104
 [<ffffffff806b2825>] start_kernel+0x245/0x2d0
 [<ffffffff806b2127>] _sinittext+0x127/0x170

*** SLUB: Freepointer corrupt in kmalloc-32@0xffff8101c0344050 Slab 0xffff8101df01a200
    offset=80 flags=0xe000000000000c7 inuse=1 freelist=0xffff8101c0344050
  Bytes b4 0xffff8101c0344040:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff8101c0344050:  91 65 28 80 ff ff ff ff 00 00 00 00 00 00 00 00 .e(.ÿÿÿÿ........
    Object 0xffff8101c0344060:  6b db fe ff 00 00 00 00 00 00 00 00 00 00 00 00 kÛþÿ............
    Object 0xffff8101c0344070:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff8101c0344080:  00 00 00 00 00 00 00 00 d8 c7 4a be 01 81 ff ff ........ØÇJ¾..ÿÿ
    Object 0xffff8101c0344090:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
FreePointer 0xffff8101c0344050 -> 0xffffffff80286591
Last alloc: 0xffff8101c03440f0 jiffies since=4294892395 cpu=0 pid=0
    Filler 0xffff8101c03440d0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101c03440e0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101c03440f0:  40 41 34 c0 01 81 ff ff 00 00 00 00 00 00 00 00 @A4À..ÿÿ........
    Filler 0xffff8101c0344100:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101c0344110:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101c0344120:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101c0344130:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101c0344140:  90 41 34 c0 01 81 ff ff 00 00 00 00 00 00 00 00 .A4À..ÿÿ........

Call Trace:
 [<ffffffff802840e5>] object_err+0x105/0x1b0
 [<ffffffff80284485>] check_object+0x1b5/0x1d0
 [<ffffffff80284ad4>] alloc_object_checks+0x64/0x110
 [<ffffffff8028611c>] kmem_cache_alloc+0xfc/0x1a0
 [<ffffffff80286591>] sysfs_slab_alias+0x31/0x80
 [<ffffffff80286afc>] kmem_cache_create+0x20c/0x220
 [<ffffffff803b76ea>] acpi_os_create_cache+0x1e/0x2e
 [<ffffffff803cc6a6>] acpi_ut_create_caches+0x88/0xae
 [<ffffffff803ce560>] acpi_ut_init_globals+0x9/0x1a1
 [<ffffffff803cda3b>] acpi_initialize_subsystem+0x3a/0x8b
 [<ffffffff806d382b>] acpi_early_init+0x5c/0x104
 [<ffffffff806b2825>] start_kernel+0x245/0x2d0
 [<ffffffff806b2127>] _sinittext+0x127/0x170

*** SLUB: Freepointer corrupt in kmalloc-32@0xffff810173f38050 Slab 0xffff81017f9f9c00
    offset=80 flags=0x2000000000000c7 inuse=1 freelist=0xffff810173f38050
  Bytes b4 0xffff810173f38040:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff810173f38050:  91 65 28 80 ff ff ff ff 00 00 00 00 00 00 00 00 .e(.ÿÿÿÿ........
    Object 0xffff810173f38060:  6c db fe ff 00 00 00 00 00 00 00 00 00 00 00 00 lÛþÿ............
    Object 0xffff810173f38070:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff810173f38080:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff810173f38090:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
FreePointer 0xffff810173f38050 -> 0xffffffff80286591
Last alloc: 0xffff810173f380f0 jiffies since=4294892396 cpu=0 pid=0
    Filler 0xffff810173f380d0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810173f380e0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810173f380f0:  40 81 f3 73 01 81 ff ff 00 00 00 00 00 00 00 00 @.ós..ÿÿ........
    Filler 0xffff810173f38100:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810173f38110:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810173f38120:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810173f38130:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810173f38140:  90 81 f3 73 01 81 ff ff 00 00 00 00 00 00 00 00 ..ós..ÿÿ........

Call Trace:
 [<ffffffff802840e5>] object_err+0x105/0x1b0
 [<ffffffff80284485>] check_object+0x1b5/0x1d0
 [<ffffffff80284ad4>] alloc_object_checks+0x64/0x110
 [<ffffffff8028611c>] kmem_cache_alloc+0xfc/0x1a0
 [<ffffffff80286591>] sysfs_slab_alias+0x31/0x80
 [<ffffffff80286afc>] kmem_cache_create+0x20c/0x220
 [<ffffffff803b76ea>] acpi_os_create_cache+0x1e/0x2e
 [<ffffffff803cc6c7>] acpi_ut_create_caches+0xa9/0xae
 [<ffffffff803ce560>] acpi_ut_init_globals+0x9/0x1a1
 [<ffffffff803cda3b>] acpi_initialize_subsystem+0x3a/0x8b
 [<ffffffff806d382b>] acpi_early_init+0x5c/0x104
 [<ffffffff806b2825>] start_kernel+0x245/0x2d0
 [<ffffffff806b2127>] _sinittext+0x127/0x170

*** SLUB: Freepointer corrupt in kmalloc-32@0xffff8101a02fd050 Slab 0xffff8101bf017e80
    offset=80 flags=0xa000000000000c7 inuse=1 freelist=0xffff8101a02fd050
  Bytes b4 0xffff8101a02fd040:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff8101a02fd050:  91 65 28 80 ff ff ff ff 00 00 00 00 00 00 00 00 .e(.ÿÿÿÿ........
    Object 0xffff8101a02fd060:  6d db fe ff 00 00 00 00 00 00 00 00 00 00 00 00 mÛþÿ............
    Object 0xffff8101a02fd070:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff8101a02fd080:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff8101a02fd090:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
FreePointer 0xffff8101a02fd050 -> 0xffffffff80286591
Last alloc: 0xffff8101a02fd0f0 jiffies since=4294892397 cpu=0 pid=0
    Filler 0xffff8101a02fd0d0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101a02fd0e0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101a02fd0f0:  40 d1 2f a0 01 81 ff ff 00 00 00 00 00 00 00 00 @Ñ/...ÿÿ........
    Filler 0xffff8101a02fd100:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101a02fd110:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101a02fd120:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101a02fd130:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101a02fd140:  90 d1 2f a0 01 81 ff ff 00 00 00 00 00 00 00 00 .Ñ/...ÿÿ........

Call Trace:
 [<ffffffff802840e5>] object_err+0x105/0x1b0
 [<ffffffff80284485>] check_object+0x1b5/0x1d0
 [<ffffffff80284ad4>] alloc_object_checks+0x64/0x110
 [<ffffffff8028611c>] kmem_cache_alloc+0xfc/0x1a0
 [<ffffffff803b7795>] acpi_os_create_semaphore+0x36/0x71
 [<ffffffff803cef18>] acpi_ut_mutex_initialize+0x33/0x77
 [<ffffffff803cda40>] acpi_initialize_subsystem+0x3f/0x8b
 [<ffffffff806d382b>] acpi_early_init+0x5c/0x104
 [<ffffffff806b2825>] start_kernel+0x245/0x2d0
 [<ffffffff806b2127>] _sinittext+0x127/0x170

*** SLUB: Freepointer corrupt in kmalloc-32@0xffff8101c0343050 Slab 0xffff8101df01a180
    offset=80 flags=0xe000000000000c7 inuse=1 freelist=0xffff8101c0343050
  Bytes b4 0xffff8101c0343040:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff8101c0343050:  95 77 3b 80 ff ff ff ff 00 00 00 00 00 00 00 00 .w;.ÿÿÿÿ........
    Object 0xffff8101c0343060:  6e db fe ff 00 00 00 00 00 00 00 00 00 00 00 00 nÛþÿ............
    Object 0xffff8101c0343070:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff8101c0343080:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff8101c0343090:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
FreePointer 0xffff8101c0343050 -> 0xffffffff803b7795
Last alloc: 0xffff8101c03430f0 jiffies since=4294892398 cpu=0 pid=0
    Filler 0xffff8101c03430d0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101c03430e0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101c03430f0:  40 31 34 c0 01 81 ff ff 00 00 00 00 00 00 00 00 @14À..ÿÿ........
    Filler 0xffff8101c0343100:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101c0343110:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101c0343120:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101c0343130:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101c0343140:  90 31 34 c0 01 81 ff ff 00 00 00 00 00 00 00 00 .14À..ÿÿ........

Call Trace:
 [<ffffffff802840e5>] object_err+0x105/0x1b0
 [<ffffffff80284485>] check_object+0x1b5/0x1d0
 [<ffffffff80284ad4>] alloc_object_checks+0x64/0x110
 [<ffffffff8028611c>] kmem_cache_alloc+0xfc/0x1a0
 [<ffffffff803b7795>] acpi_os_create_semaphore+0x36/0x71
 [<ffffffff803cef18>] acpi_ut_mutex_initialize+0x33/0x77
 [<ffffffff803cda40>] acpi_initialize_subsystem+0x3f/0x8b
 [<ffffffff806d382b>] acpi_early_init+0x5c/0x104
 [<ffffffff806b2825>] start_kernel+0x245/0x2d0
 [<ffffffff806b2127>] _sinittext+0x127/0x170

*** SLUB: Freepointer corrupt in kmalloc-32@0xffff810173f37050 Slab 0xffff81017f9f9b80
    offset=80 flags=0x2000000000000c7 inuse=1 freelist=0xffff810173f37050
  Bytes b4 0xffff810173f37040:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff810173f37050:  95 77 3b 80 ff ff ff ff 00 00 00 00 00 00 00 00 .w;.ÿÿÿÿ........
    Object 0xffff810173f37060:  6f db fe ff 00 00 00 00 00 00 00 00 00 00 00 00 oÛþÿ............
    Object 0xffff810173f37070:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff810173f37080:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff810173f37090:  18 81 70 72 01 81 ff ff 00 00 00 00 00 00 00 00 ..pr..ÿÿ........
FreePointer 0xffff810173f37050 -> 0xffffffff803b7795
Last alloc: 0xffff810173f370f0 jiffies since=4294892399 cpu=0 pid=0
    Filler 0xffff810173f370d0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810173f370e0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810173f370f0:  40 71 f3 73 01 81 ff ff 00 00 00 00 00 00 00 00 @qós..ÿÿ........
    Filler 0xffff810173f37100:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810173f37110:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810173f37120:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810173f37130:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810173f37140:  90 71 f3 73 01 81 ff ff 00 00 00 00 00 00 00 00 .qós..ÿÿ........

Call Trace:
 [<ffffffff802840e5>] object_err+0x105/0x1b0
 [<ffffffff80284485>] check_object+0x1b5/0x1d0
 [<ffffffff80284ad4>] alloc_object_checks+0x64/0x110
 [<ffffffff8028611c>] kmem_cache_alloc+0xfc/0x1a0
 [<ffffffff803b7795>] acpi_os_create_semaphore+0x36/0x71
 [<ffffffff803cef18>] acpi_ut_mutex_initialize+0x33/0x77
 [<ffffffff803cda40>] acpi_initialize_subsystem+0x3f/0x8b
 [<ffffffff806d382b>] acpi_early_init+0x5c/0x104
 [<ffffffff806b2825>] start_kernel+0x245/0x2d0
 [<ffffffff806b2127>] _sinittext+0x127/0x170

*** SLUB: Freepointer corrupt in kmalloc-32@0xffff81018033c050 Slab 0xffff81019f019e00
    offset=80 flags=0x6000000000000c7 inuse=1 freelist=0xffff81018033c050
  Bytes b4 0xffff81018033c040:  40 c0 33 80 01 81 ff ff 40 c0 33 80 01 81 ff ff @À3...ÿÿ@À3...ÿÿ
    Object 0xffff81018033c050:  95 77 3b 80 ff ff ff ff 00 00 00 00 00 00 00 00 .w;.ÿÿÿÿ........
    Object 0xffff81018033c060:  70 db fe ff 00 00 00 00 00 00 00 00 00 00 00 00 pÛþÿ............
    Object 0xffff81018033c070:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff81018033c080:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff81018033c090:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
FreePointer 0xffff81018033c050 -> 0xffffffff803b7795
Last alloc: 0xffff81018033c0f0 jiffies since=4294892400 cpu=0 pid=0
    Filler 0xffff81018033c0d0:  00 00 00 00 00 00 00 00 80 44 f1 73 01 81 ff ff .........Dñs..ÿÿ
    Filler 0xffff81018033c0e0:  30 80 f1 73 01 81 ff ff c0 60 f1 73 01 81 ff ff 0.ñs..ÿÿÀ`ñs..ÿÿ
    Filler 0xffff81018033c0f0:  40 c1 33 80 01 81 ff ff 00 00 00 00 00 00 00 00 @Á3...ÿÿ........
    Filler 0xffff81018033c100:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff81018033c110:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff81018033c120:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff81018033c130:  00 00 00 00 00 00 00 00 a7 1d 14 46 00 00 00 00 ........§..F....
    Filler 0xffff81018033c140:  90 c1 33 80 01 81 ff ff 00 00 00 00 00 00 00 00 .Á3...ÿÿ........

Call Trace:
 [<ffffffff802840e5>] object_err+0x105/0x1b0
 [<ffffffff80284485>] check_object+0x1b5/0x1d0
 [<ffffffff80284ad4>] alloc_object_checks+0x64/0x110
 [<ffffffff8028611c>] kmem_cache_alloc+0xfc/0x1a0
 [<ffffffff803b7795>] acpi_os_create_semaphore+0x36/0x71
 [<ffffffff803cef18>] acpi_ut_mutex_initialize+0x33/0x77
 [<ffffffff803cda40>] acpi_initialize_subsystem+0x3f/0x8b
 [<ffffffff806d382b>] acpi_early_init+0x5c/0x104
 [<ffffffff806b2825>] start_kernel+0x245/0x2d0
 [<ffffffff806b2127>] _sinittext+0x127/0x170

*** SLUB: Freepointer corrupt in kmalloc-32@0xffff8101a0342050 Slab 0xffff8101bf01a100
    offset=80 flags=0xa000000000000c7 inuse=1 freelist=0xffff8101a0342050
  Bytes b4 0xffff8101a0342040:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff8101a0342050:  95 77 3b 80 ff ff ff ff 00 00 00 00 00 00 00 00 .w;.ÿÿÿÿ........
    Object 0xffff8101a0342060:  71 db fe ff 00 00 00 00 00 00 00 00 00 00 00 00 qÛþÿ............
    Object 0xffff8101a0342070:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff8101a0342080:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff8101a0342090:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
FreePointer 0xffff8101a0342050 -> 0xffffffff803b7795
Last alloc: 0xffff8101a03420f0 jiffies since=4294892401 cpu=0 pid=0
    Filler 0xffff8101a03420d0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101a03420e0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101a03420f0:  40 21 34 a0 01 81 ff ff 00 00 00 00 00 00 00 00 @!4...ÿÿ........
    Filler 0xffff8101a0342100:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101a0342110:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101a0342120:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101a0342130:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101a0342140:  90 21 34 a0 01 81 ff ff 00 00 00 00 00 00 00 00 .!4...ÿÿ........

Call Trace:
 [<ffffffff802840e5>] object_err+0x105/0x1b0
 [<ffffffff80284485>] check_object+0x1b5/0x1d0
 [<ffffffff80284ad4>] alloc_object_checks+0x64/0x110
 [<ffffffff8028611c>] kmem_cache_alloc+0xfc/0x1a0
 [<ffffffff803b7795>] acpi_os_create_semaphore+0x36/0x71
 [<ffffffff803cef18>] acpi_ut_mutex_initialize+0x33/0x77
 [<ffffffff803cda40>] acpi_initialize_subsystem+0x3f/0x8b
 [<ffffffff806d382b>] acpi_early_init+0x5c/0x104
 [<ffffffff806b2825>] start_kernel+0x245/0x2d0
 [<ffffffff806b2127>] _sinittext+0x127/0x170

*** SLUB: Freepointer corrupt in kmalloc-32@0xffff8101c0342050 Slab 0xffff8101df01a100
    offset=80 flags=0xe000000000000c7 inuse=1 freelist=0xffff8101c0342050
  Bytes b4 0xffff8101c0342040:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff8101c0342050:  95 77 3b 80 ff ff ff ff 00 00 00 00 00 00 00 00 .w;.ÿÿÿÿ........
    Object 0xffff8101c0342060:  72 db fe ff 00 00 00 00 00 00 00 00 00 00 00 00 rÛþÿ............
    Object 0xffff8101c0342070:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff8101c0342080:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff8101c0342090:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
FreePointer 0xffff8101c0342050 -> 0xffffffff803b7795
Last alloc: 0xffff8101c03420f0 jiffies since=4294892402 cpu=0 pid=0
    Filler 0xffff8101c03420d0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101c03420e0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101c03420f0:  40 21 34 c0 01 81 ff ff 00 00 00 00 00 00 00 00 @!4À..ÿÿ........
    Filler 0xffff8101c0342100:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101c0342110:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101c0342120:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101c0342130:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff8101c0342140:  90 21 34 c0 01 81 ff ff 00 00 00 00 00 00 00 00 .!4À..ÿÿ........

Call Trace:
 [<ffffffff802840e5>] object_err+0x105/0x1b0
 [<ffffffff80284485>] check_object+0x1b5/0x1d0
 [<ffffffff80284ad4>] alloc_object_checks+0x64/0x110
 [<ffffffff8028611c>] kmem_cache_alloc+0xfc/0x1a0
 [<ffffffff803b7795>] acpi_os_create_semaphore+0x36/0x71
 [<ffffffff803cef18>] acpi_ut_mutex_initialize+0x33/0x77
 [<ffffffff803cda40>] acpi_initialize_subsystem+0x3f/0x8b
 [<ffffffff806d382b>] acpi_early_init+0x5c/0x104
 [<ffffffff806b2825>] start_kernel+0x245/0x2d0
 [<ffffffff806b2127>] _sinittext+0x127/0x170

*** SLUB: Freepointer corrupt in kmalloc-32@0xffff810173f36050 Slab 0xffff81017f9f9b00
    offset=80 flags=0x2000000000000c7 inuse=1 freelist=0xffff810173f36050
  Bytes b4 0xffff810173f36040:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff810173f36050:  95 77 3b 80 ff ff ff ff 00 00 00 00 00 00 00 00 .w;.ÿÿÿÿ........
    Object 0xffff810173f36060:  73 db fe ff 00 00 00 00 00 00 00 00 00 00 00 00 sÛþÿ............
    Object 0xffff810173f36070:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff810173f36080:  00 00 00 00 00 00 00 00 18 97 6f bd 01 81 ff ff ..........o½..ÿÿ
    Object 0xffff810173f36090:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
FreePointer 0xffff810173f36050 -> 0xffffffff803b7795
Last alloc: 0xffff810173f360f0 jiffies since=4294892403 cpu=0 pid=0
    Filler 0xffff810173f360d0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810173f360e0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810173f360f0:  40 61 f3 73 01 81 ff ff 00 00 00 00 00 00 00 00 @aós..ÿÿ........
    Filler 0xffff810173f36100:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810173f36110:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810173f36120:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810173f36130:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff810173f36140:  90 61 f3 73 01 81 ff ff 00 00 00 00 00 00 00 00 .aós..ÿÿ........

Call Trace:
 [<ffffffff802840e5>] object_err+0x105/0x1b0
 [<ffffffff80284485>] check_object+0x1b5/0x1d0
 [<ffffffff80284ad4>] alloc_object_checks+0x64/0x110
 [<ffffffff8028611c>] kmem_cache_alloc+0xfc/0x1a0
 [<ffffffff803b7795>] acpi_os_create_semaphore+0x36/0x71
 [<ffffffff803cef18>] acpi_ut_mutex_initialize+0x33/0x77
 [<ffffffff803cda40>] acpi_initialize_subsystem+0x3f/0x8b
 [<ffffffff806d382b>] acpi_early_init+0x5c/0x104
 [<ffffffff806b2825>] start_kernel+0x245/0x2d0
 [<ffffffff806b2127>] _sinittext+0x127/0x170

*** SLUB: Freepointer corrupt in kmalloc-32@0xffff81018033b050 Slab 0xffff81019f019d80
    offset=80 flags=0x6000000000000c7 inuse=1 freelist=0xffff81018033b050
  Bytes b4 0xffff81018033b040:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff81018033b050:  95 77 3b 80 ff ff ff ff 00 00 00 00 00 00 00 00 .w;.ÿÿÿÿ........
    Object 0xffff81018033b060:  74 db fe ff 00 00 00 00 00 00 00 00 00 00 00 00 tÛþÿ............
    Object 0xffff81018033b070:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff81018033b080:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Object 0xffff81018033b090:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
FreePointer 0xffff81018033b050 -> 0xffffffff803b7795
Last alloc: 0xffff81018033b0f0 jiffies since=4294892404 cpu=0 pid=0
    Filler 0xffff81018033b0d0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff81018033b0e0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff81018033b0f0:  40 b1 33 80 01 81 ff ff 00 00 00 00 00 00 00 00 @±3...ÿÿ........
    Filler 0xffff81018033b100:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff81018033b110:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff81018033b120:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff81018033b130:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    Filler 0xffff81018033b140:  90 b1 33 80 01 81 ff ff 00 00 00 00 00 00 00 00 .±3...ÿÿ........

Call Trace:
 [<ffffffff802840e5>] object_err+0x105/0x1b0
 [<ffffffff80284485>] check_object+0x1b5/0x1d0
 [<ffffffff80284ad4>] alloc_object_checks+0x64/0x110
 [<ffffffff8028611c>] kmem_cache_alloc+0xfc/0x1a0
 [<ffffffff803b7795>] acpi_os_create_semaphore+0x36/0x71
 [<ffffffff803cef18>] acpi_ut_mutex_initialize+0x33/0x77
 [<ffffffff803cda40>] acpi_initialize_subsystem+0x3f/0x8b
 [<ffffffff806d382b>] acpi_early_init+0x5c/0x104
 [<ffffffff806b2825>] start_kernel+0x245/0x2d0
 [<ffffffff806b2127>] _sinittext+0x127/0x170

Re: 2.6.21-rc5-mm4 (SLUB)

[Christoph Lameter]

On Wed, 4 Apr 2007, Badari Pulavarty wrote:

> Here is the slub_debug=FU output with the above patch.

Hmmm... Looks like the object is actually free. Someone writes beyond the 
end of the earlier object. Setting Z should check overwrites but it 
switched off merging. So set

slub_debug = FZ

Analoguos to the last patch you would need to take out redzoning from 
the flags that stop merging. Then rerun. Maybe we can track it down this 
way.

Hmmm... Maybe remove all the debug flags from those that avoid merging and 
then run with full debug. That should theoretically flush it out.
 

Re: 2.6.21-rc5-mm4 (SLUB)

[Badari Pulavarty]

On Wed, 2007-04-04 at 15:59 -0700, Christoph Lameter wrote:
> On Wed, 4 Apr 2007, Badari Pulavarty wrote:
> 
> > Here is the slub_debug=FU output with the above patch.
> 
> Hmmm... Looks like the object is actually free. Someone writes beyond the 
> end of the earlier object. Setting Z should check overwrites but it 
> switched off merging. So set
> 
> slub_debug = FZ
> 
> Analoguos to the last patch you would need to take out redzoning from 
> the flags that stop merging. Then rerun. Maybe we can track it down this 
> way.

Hmm.. I did that and machine boots fine, with absolutely no
debug messages :(

Thanks,
Badari

Re: 2.6.21-rc5-mm4 (SLUB)

[Christoph Lameter]

On Wed, 4 Apr 2007, Badari Pulavarty wrote:

> On Wed, 2007-04-04 at 15:59 -0700, Christoph Lameter wrote:
> > On Wed, 4 Apr 2007, Badari Pulavarty wrote:
> > 
> > > Here is the slub_debug=FU output with the above patch.
> > 
> > Hmmm... Looks like the object is actually free. Someone writes beyond the 
> > end of the earlier object. Setting Z should check overwrites but it 
> > switched off merging. So set
> > 
> > slub_debug = FZ
> > 
> > Analoguos to the last patch you would need to take out redzoning from 
> > the flags that stop merging. Then rerun. Maybe we can track it down this 
> > way.
> 
> Hmm.. I did that and machine boots fine, with absolutely no
> debug messages :(

Were the slabs merged? Look at /sys/slab and see if there are any symlinks 
there.

Re: 2.6.21-rc5-mm4 (SLUB)

[Badari Pulavarty]

On Wed, 2007-04-04 at 17:31 -0700, Christoph Lameter wrote:
> On Wed, 4 Apr 2007, Badari Pulavarty wrote:
> 
> > On Wed, 2007-04-04 at 15:59 -0700, Christoph Lameter wrote:
> > > On Wed, 4 Apr 2007, Badari Pulavarty wrote:
> > > 
> > > > Here is the slub_debug=FU output with the above patch.
> > > 
> > > Hmmm... Looks like the object is actually free. Someone writes beyond the 
> > > end of the earlier object. Setting Z should check overwrites but it 
> > > switched off merging. So set
> > > 
> > > slub_debug = FZ
> > > 
> > > Analoguos to the last patch you would need to take out redzoning from 
> > > the flags that stop merging. Then rerun. Maybe we can track it down this 
> > > way.
> > 
> > Hmm.. I did that and machine boots fine, with absolutely no
> > debug messages :(
> 
> Were the slabs merged? Look at /sys/slab and see if there are any symlinks 
> there.
> 

elm3b29:/sys/slab # ls -ltr
total 0
drwxr-xr-x 2 root root 0 Apr  4 17:40 sock_inode_cache
drwxr-xr-x 2 root root 0 Apr  4 17:40 skbuff_fclone_cache
drwxr-xr-x 2 root root 0 Apr  4 17:40 sigqueue
drwxr-xr-x 2 root root 0 Apr  4 17:40 shmem_inode_cache
drwxr-xr-x 2 root root 0 Apr  4 17:40 radix_tree_node
drwxr-xr-x 2 root root 0 Apr  4 17:40 proc_inode_cache
drwxr-xr-x 2 root root 0 Apr  4 17:40 ip_dst_cache
drwxr-xr-x 2 root root 0 Apr  4 17:40 file_lock_cache
drwxr-xr-x 2 root root 0 Apr  4 17:40 blkdev_requests
drwxr-xr-x 2 root root 0 Apr  4 17:40 blkdev_queue
drwxr-xr-x 2 root root 0 Apr  4 17:40 blkdev_ioc
drwxr-xr-x 2 root root 0 Apr  4 17:40 biovec-64
drwxr-xr-x 2 root root 0 Apr  4 17:40 biovec-256
drwxr-xr-x 2 root root 0 Apr  4 17:40 biovec-128
drwxr-xr-x 2 root root 0 Apr  4 17:40 bdev_cache
drwxr-xr-x 2 root root 0 Apr  4 17:40 TCP
drwxr-xr-x 2 root root 0 Apr  4 17:40 Acpi-State
drwxr-xr-x 2 root root 0 Apr  4 17:40 Acpi-ParseExt
drwxr-xr-x 2 root root 0 Apr  4 17:40 Acpi-Operand
drwxr-xr-x 2 root root 0 Apr  4 17:40 Acpi-Namespace
drwxr-xr-x 2 root root 0 Apr  4 17:40 vm_area_struct
drwxr-xr-x 2 root root 0 Apr  4 17:40 task_struct
drwxr-xr-x 2 root root 0 Apr  4 17:40 sysfs_dir_cache
drwxr-xr-x 2 root root 0 Apr  4 17:40 signal_cache
drwxr-xr-x 2 root root 0 Apr  4 17:40 sighand_cache
drwxr-xr-x 2 root root 0 Apr  4 17:40 pid
drwxr-xr-x 2 root root 0 Apr  4 17:40 names_cache
drwxr-xr-x 2 root root 0 Apr  4 17:40 mm_struct
drwxr-xr-x 2 root root 0 Apr  4 17:40 kmem_cache_node
drwxr-xr-x 2 root root 0 Apr  4 17:40 kmalloc-96
drwxr-xr-x 2 root root 0 Apr  4 17:40 kmalloc-8192
drwxr-xr-x 2 root root 0 Apr  4 17:40 kmalloc-8
drwxr-xr-x 2 root root 0 Apr  4 17:40 kmalloc-65536
drwxr-xr-x 2 root root 0 Apr  4 17:40 kmalloc-64
drwxr-xr-x 2 root root 0 Apr  4 17:40 kmalloc-512
drwxr-xr-x 2 root root 0 Apr  4 17:40 kmalloc-4096
drwxr-xr-x 2 root root 0 Apr  4 17:40 kmalloc-32768
drwxr-xr-x 2 root root 0 Apr  4 17:40 kmalloc-32
drwxr-xr-x 2 root root 0 Apr  4 17:40 kmalloc-262144
drwxr-xr-x 2 root root 0 Apr  4 17:40 kmalloc-256
drwxr-xr-x 2 root root 0 Apr  4 17:40 kmalloc-2048
drwxr-xr-x 2 root root 0 Apr  4 17:40 kmalloc-192
drwxr-xr-x 2 root root 0 Apr  4 17:40 kmalloc-16384
drwxr-xr-x 2 root root 0 Apr  4 17:40 kmalloc-16
drwxr-xr-x 2 root root 0 Apr  4 17:40 kmalloc-131072
drwxr-xr-x 2 root root 0 Apr  4 17:40 kmalloc-128
drwxr-xr-x 2 root root 0 Apr  4 17:40 kmalloc-1024
drwxr-xr-x 2 root root 0 Apr  4 17:40 inode_cache
drwxr-xr-x 2 root root 0 Apr  4 17:40 idr_layer_cache
drwxr-xr-x 2 root root 0 Apr  4 17:40 fs_cache
drwxr-xr-x 2 root root 0 Apr  4 17:40 filp
drwxr-xr-x 2 root root 0 Apr  4 17:40 dentry_cache
drwxr-xr-x 2 root root 0 Apr  4 17:40 buffer_head
drwxr-xr-x 2 root root 0 Apr  4 17:40 anon_vma
drwxr-xr-x 2 root root 0 Apr  4 17:40 dquot
drwxr-xr-x 2 root root 0 Apr  4 17:40 reiser_inode_cache
drwxr-xr-x 2 root root 0 Apr  4 17:40 nfs_inode_cache
drwxr-xr-x 2 root root 0 Apr  4 17:40 nfs_direct_cache
drwxr-xr-x 2 root root 0 Apr  4 17:40 mqueue_inode_cache
drwxr-xr-x 2 root root 0 Apr  4 17:40 minix_inode_cache
drwxr-xr-x 2 root root 0 Apr  4 17:40 journal_head
drwxr-xr-x 2 root root 0 Apr  4 17:40 isofs_inode_cache
drwxr-xr-x 2 root root 0 Apr  4 17:40 hugetlbfs_inode_cache
drwxr-xr-x 2 root root 0 Apr  4 17:40 ext3_xattr
drwxr-xr-x 2 root root 0 Apr  4 17:40 ext3_inode_cache
drwxr-xr-x 2 root root 0 Apr  4 17:40 ext2_xattr
drwxr-xr-x 2 root root 0 Apr  4 17:40 ext2_inode_cache
drwxr-xr-x 2 root root 0 Apr  4 17:40 cfq_pool
drwxr-xr-x 2 root root 0 Apr  4 17:40 cfq_ioc_pool
drwxr-xr-x 2 root root 0 Apr  4 17:40 UNIX
drwxr-xr-x 2 root root 0 Apr  4 17:40 rpc_inode_cache
drwxr-xr-x 2 root root 0 Apr  4 17:40 rpc_buffers
drwxr-xr-x 2 root root 0 Apr  4 17:40 revokefs_inode_cache
drwxr-xr-x 2 root root 0 Apr  4 17:40 TCPv6
drwxr-xr-x 2 root root 0 Apr  4 17:54 fat_inode_cache
drwxr-xr-x 2 root root 0 Apr  4 17:54 fat_cache
drwxr-xr-x 2 root root 0 Apr  4 17:54 sgpool-64
drwxr-xr-x 2 root root 0 Apr  4 17:54 sgpool-32
drwxr-xr-x 2 root root 0 Apr  4 17:54 sgpool-128
drwxr-xr-x 2 root root 0 Apr  4 17:54 scsi_io_context
drwxr-xr-x 2 root root 0 Apr  4 17:55 jbd_4k
lrwxrwxrwx 1 root root 0 Apr  4 20:46 scsi_cmd_cache -> ../slab/ip_dst_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 xfrm_dst_cache -> ../slab/ip_dst_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 utrace_engine_cache -> ../slab/fs_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 utrace_cache -> ../slab/fs_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 uid_cache -> ../slab/fs_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 tw_sock_TCPv6 -> ../slab/filp
lrwxrwxrwx 1 root root 0 Apr  4 20:46 tw_sock_TCP -> ../slab/filp
lrwxrwxrwx 1 root root 0 Apr  4 20:46 tcp_bind_bucket -> ../slab/fs_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 skbuff_head_cache -> ../slab/filp
lrwxrwxrwx 1 root root 0 Apr  4 20:46 shared_policy_node -> ../slab/kmem_cache_node
lrwxrwxrwx 1 root root 0 Apr  4 20:46 sgpool-8 -> ../slab/filp
lrwxrwxrwx 1 root root 0 Apr  4 20:46 sgpool-16 -> ../slab/skbuff_fclone_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 secpath_cache -> ../slab/fs_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 rpc_tasks -> ../slab/ip_dst_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 revoke_table -> ../slab/kmalloc-8
lrwxrwxrwx 1 root root 0 Apr  4 20:46 revoke_record -> ../slab/fs_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 request_sock_TCPv6 -> ../slab/filp
lrwxrwxrwx 1 root root 0 Apr  4 20:46 request_sock_TCP -> ../slab/fs_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 qla2xxx_srbs -> ../slab/fs_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 posix_timers_cache -> ../slab/kmalloc-128
lrwxrwxrwx 1 root root 0 Apr  4 20:46 numa_policy -> ../slab/kmalloc-16
lrwxrwxrwx 1 root root 0 Apr  4 20:46 nfs_write_data -> ../slab/signal_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 nfs_read_data -> ../slab/signal_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 nfs_page -> ../slab/fs_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 ndisc_cache -> ../slab/filp
lrwxrwxrwx 1 root root 0 Apr  4 20:46 mnt_cache -> ../slab/filp
lrwxrwxrwx 1 root root 0 Apr  4 20:46 kioctx -> ../slab/ip_dst_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 kiocb -> ../slab/filp
lrwxrwxrwx 1 root root 0 Apr  4 20:46 journal_handle -> ../slab/kmalloc-16
lrwxrwxrwx 1 root root 0 Apr  4 20:46 ip_mrt_cache -> ../slab/fs_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 ip_fib_hash -> ../slab/fs_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 ip_fib_alias -> ../slab/fs_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 ip6_dst_cache -> ../slab/ip_dst_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 inotify_watch_cache -> ../slab/Acpi-Operand
lrwxrwxrwx 1 root root 0 Apr  4 20:46 inotify_event_cache -> ../slab/Acpi-Namespace
lrwxrwxrwx 1 root root 0 Apr  4 20:46 inet_peer_cache -> ../slab/fs_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 flow_cache -> ../slab/fs_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 files_cache -> ../slab/signal_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 fib6_nodes -> ../slab/fs_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 fasync_cache -> ../slab/kmalloc-16
lrwxrwxrwx 1 root root 0 Apr  4 20:46 eventpoll_pwq -> ../slab/Acpi-Operand
lrwxrwxrwx 1 root root 0 Apr  4 20:46 eventpoll_epi -> ../slab/fs_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 dnotify_cache -> ../slab/Acpi-Namespace
lrwxrwxrwx 1 root root 0 Apr  4 20:46 dm_tio -> ../slab/kmalloc-16
lrwxrwxrwx 1 root root 0 Apr  4 20:46 dm_io -> ../slab/Acpi-Namespace
lrwxrwxrwx 1 root root 0 Apr  4 20:46 configfs_dir_cache -> ../slab/Acpi-State
lrwxrwxrwx 1 root root 0 Apr  4 20:46 biovec-4 -> ../slab/fs_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 biovec-16 -> ../slab/filp
lrwxrwxrwx 1 root root 0 Apr  4 20:46 biovec-1 -> ../slab/fs_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 bio -> ../slab/fs_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 arp_cache -> ../slab/filp
lrwxrwxrwx 1 root root 0 Apr  4 20:46 UDPv6 -> ../slab/mm_struct
lrwxrwxrwx 1 root root 0 Apr  4 20:46 UDPLITEv6 -> ../slab/mm_struct
lrwxrwxrwx 1 root root 0 Apr  4 20:46 UDP-Lite -> ../slab/signal_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 UDP -> ../slab/signal_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 RAWv6 -> ../slab/mm_struct
lrwxrwxrwx 1 root root 0 Apr  4 20:46 RAW -> ../slab/signal_cache
lrwxrwxrwx 1 root root 0 Apr  4 20:46 Acpi-Parse -> ../slab/Acpi-Namespace

Re: 2.6.21-rc5-mm4 (SLUB)

[Christoph Lameter]

On Wed, 4 Apr 2007, Badari Pulavarty wrote:

> > Were the slabs merged? Look at /sys/slab and see if there are any symlinks 
> > there.
> > 

Ok. symlinks there. Its a sporadic thing. I think I am going to add a slab
validator to SLUB that goes through all slabs and checks all objects for 
validity. Then we can trigger a scan through the acpi caches which should 
locate the problem.

Re: 2.6.21-rc5-mm4 (SLUB)

[Christoph Lameter]

Here is a patch that adds validation (only for cpuslabs and partial 
slabs but thats where the action is). Apply this patch
and then do

echo 1 >/sys/slab/<cache-to-check>/validate

I suggest to boot with full debugging and then run this on the ACPI slabs.

Signed-off-by: Christoph Lameter <clameter@sgi.com>

Index: linux-2.6.21-rc5-mm4/mm/slub.c
===================================================================
--- linux-2.6.21-rc5-mm4.orig/mm/slub.c	2007-04-04 20:26:03.000000000 -0700
+++ linux-2.6.21-rc5-mm4/mm/slub.c	2007-04-04 21:26:15.000000000 -0700
@@ -2280,6 +2280,67 @@ void *__kmalloc_node_track_caller(size_t
 
 #ifdef CONFIG_SYSFS
 
+static int validate_slab(struct kmem_cache *s, struct page *page)
+{
+	void *p;
+	void *addr = page_address(page);
+	unsigned long map[BITS_TO_LONGS(s->objects)];
+
+	if (!check_slab(s, page) ||
+			!on_freelist(s, page, NULL))
+		return 0;
+
+	/* Now we know that a valid freelist exists */
+	bitmap_zero(map, s->objects);
+
+	for(p = page->freelist; p; p = get_freepointer(s, p)) {
+		set_bit((p - addr) / s->size, map);
+		if (!check_object(s, page, p, 0))
+			return 0;
+	}
+
+	for(p = addr; p < addr + s->objects * s->size; p += s->size)
+		if (!test_bit((p - addr) / s->size, map))
+			if (!check_object(s, page, p, 1))
+				return 0;
+	return 1;
+}
+
+static int validate_slab_node(struct kmem_cache *s, struct kmem_cache_node *n)
+{
+	int count = 0;
+	struct page *page;
+	unsigned long flags;
+
+	spin_lock_irqsave(&n->list_lock, flags);
+	list_for_each_entry(page, &n->partial, lru) {
+		if (slab_trylock(page)) {
+			validate_slab(s, page);
+			slab_unlock(page);
+		} else
+			printk(KERN_INFO "Skipped busy slab %p\n", page);
+		count++;
+	}
+	spin_unlock_irqrestore(&n->list_lock, flags);
+	return count;
+}
+
+static void validate_slab_cache(struct kmem_cache *s)
+{
+	int node;
+	int count = 0;
+
+	printk(KERN_INFO "--- Validating slabcache '%s'\n", s->name);
+	flush_all(s);
+	for_each_online_node(node) {
+		struct kmem_cache_node *n = get_node(s, node);
+
+		count += validate_slab_node(s, n);
+	}
+	printk(KERN_INFO "--- Checked %d slabs in '%s'\n",
+		count, s->name);
+}
+
 static unsigned long count_partial(struct kmem_cache_node *n)
 {
 	unsigned long flags;
@@ -2402,7 +2463,6 @@ struct slab_attribute {
 	static struct slab_attribute _name##_attr =  \
 	__ATTR(_name, 0644, _name##_show, _name##_store)
 
-
 static ssize_t slab_size_show(struct kmem_cache *s, char *buf)
 {
 	return sprintf(buf, "%d\n", s->size);
@@ -2609,6 +2669,22 @@ static ssize_t store_user_store(struct k
 }
 SLAB_ATTR(store_user);
 
+static ssize_t validate_show(struct kmem_cache *s, char *buf)
+{
+	return 0;
+}
+
+static ssize_t validate_store(struct kmem_cache *s,
+			const char *buf, size_t length)
+{
+	if (buf[0] == '1')
+		validate_slab_cache(s);
+	else
+		return -EINVAL;
+	return length;
+}
+SLAB_ATTR(validate);
+
 #ifdef CONFIG_NUMA
 static ssize_t defrag_ratio_show(struct kmem_cache *s, char *buf)
 {
@@ -2648,6 +2724,7 @@ static struct attribute * slab_attrs[] =
 	&red_zone_attr.attr,
 	&poison_attr.attr,
 	&store_user_attr.attr,
+	&validate_attr.attr,
 #ifdef CONFIG_ZONE_DMA
 	&cache_dma_attr.attr,
 #endif

Re: 2.6.21-rc5-mm4 (SLUB)

[Badari Pulavarty]

On Wed, 2007-04-04 at 21:29 -0700, Christoph Lameter wrote:
> Here is a patch that adds validation (only for cpuslabs and partial 
> slabs but thats where the action is). Apply this patch
> and then do
> 
> echo 1 >/sys/slab/<cache-to-check>/validate
> 
> I suggest to boot with full debugging and then run this on the ACPI slabs.

Did this and didn't trigger any problems.

(Just to be clear, booted with "slub_debug" with all the patches
applied).

--- Validating slabcache 'Acpi-Namespace'
--- Checked 0 slabs in 'Acpi-Namespace'
--- Validating slabcache 'Acpi-Operand'
--- Checked 5 slabs in 'Acpi-Operand'
--- Validating slabcache 'Acpi-Parse'
--- Checked 0 slabs in 'Acpi-Parse'
--- Validating slabcache 'Acpi-ParseExt'
--- Checked 0 slabs in 'Acpi-ParseExt'
--- Validating slabcache 'Acpi-State'
--- Checked 0 slabs in 'Acpi-State'
--- Validating slabcache 'Acpi-Namespace'
--- Checked 0 slabs in 'Acpi-Namespace'
--- Validating slabcache 'Acpi-Operand'
--- Checked 5 slabs in 'Acpi-Operand'
--- Validating slabcache 'Acpi-Parse'
--- Checked 0 slabs in 'Acpi-Parse'
--- Validating slabcache 'Acpi-ParseExt'
--- Checked 0 slabs in 'Acpi-ParseExt'
--- Validating slabcache 'Acpi-State'
--- Checked 0 slabs in 'Acpi-State'
--- Validating slabcache 'RAW'
--- Checked 1 slabs in 'RAW'
--- Validating slabcache 'RAWv6'
--- Checked 1 slabs in 'RAWv6'
--- Validating slabcache 'TCP'
--- Checked 3 slabs in 'TCP'
--- Validating slabcache 'TCPv6'
--- Checked 4 slabs in 'TCPv6'
--- Validating slabcache 'UDP-Lite'
--- Checked 0 slabs in 'UDP-Lite'
--- Validating slabcache 'UDP'
--- Checked 2 slabs in 'UDP'
--- Validating slabcache 'UDPLITEv6'
--- Checked 0 slabs in 'UDPLITEv6'
--- Validating slabcache 'UDPv6'
--- Checked 0 slabs in 'UDPv6'
--- Validating slabcache 'UNIX'
--- Checked 4 slabs in 'UNIX'
--- Validating slabcache 'anon_vma'
--- Checked 12 slabs in 'anon_vma'
--- Validating slabcache 'arp_cache'
--- Checked 2 slabs in 'arp_cache'
--- Validating slabcache 'bdev_cache'
--- Checked 3 slabs in 'bdev_cache'
--- Validating slabcache 'bio'
--- Checked 0 slabs in 'bio'
--- Validating slabcache 'biovec-1'
--- Checked 1 slabs in 'biovec-1'
--- Validating slabcache 'biovec-128'
--- Checked 1 slabs in 'biovec-128'
--- Validating slabcache 'biovec-16'
--- Checked 1 slabs in 'biovec-16'
--- Validating slabcache 'biovec-256'
--- Checked 1 slabs in 'biovec-256'
--- Validating slabcache 'biovec-4'
--- Checked 1 slabs in 'biovec-4'
--- Validating slabcache 'biovec-64'
--- Checked 1 slabs in 'biovec-64'
--- Validating slabcache 'blkdev_ioc'
--- Checked 4 slabs in 'blkdev_ioc'
--- Validating slabcache 'blkdev_queue'
--- Checked 1 slabs in 'blkdev_queue'
--- Validating slabcache 'blkdev_requests'
--- Checked 2 slabs in 'blkdev_requests'
--- Validating slabcache 'buffer_head'
--- Checked 4 slabs in 'buffer_head'
--- Validating slabcache 'cfq_ioc_pool'
--- Checked 4 slabs in 'cfq_ioc_pool'
--- Validating slabcache 'cfq_pool'
--- Checked 4 slabs in 'cfq_pool'
--- Validating slabcache 'configfs_dir_cache'
--- Checked 0 slabs in 'configfs_dir_cache'
--- Validating slabcache 'dentry_cache'
--- Checked 5 slabs in 'dentry_cache'
--- Validating slabcache 'dm_io'
--- Checked 0 slabs in 'dm_io'
--- Validating slabcache 'dm_tio'
--- Checked 0 slabs in 'dm_tio'
--- Validating slabcache 'dnotify_cache'
--- Checked 1 slabs in 'dnotify_cache'
--- Validating slabcache 'dquot'
--- Checked 0 slabs in 'dquot'
--- Validating slabcache 'eventpoll_epi'
--- Checked 1 slabs in 'eventpoll_epi'
--- Validating slabcache 'eventpoll_pwq'
--- Checked 1 slabs in 'eventpoll_pwq'
--- Validating slabcache 'ext2_inode_cache'
--- Checked 0 slabs in 'ext2_inode_cache'
--- Validating slabcache 'ext2_xattr'
--- Checked 0 slabs in 'ext2_xattr'
--- Validating slabcache 'ext3_inode_cache'
--- Checked 0 slabs in 'ext3_inode_cache'
--- Validating slabcache 'ext3_xattr'
--- Checked 0 slabs in 'ext3_xattr'
--- Validating slabcache 'fasync_cache'
--- Checked 0 slabs in 'fasync_cache'
--- Validating slabcache 'fib6_nodes'
--- Checked 1 slabs in 'fib6_nodes'
--- Validating slabcache 'file_lock_cache'
--- Checked 2 slabs in 'file_lock_cache'
--- Validating slabcache 'files_cache'
--- Checked 10 slabs in 'files_cache'
--- Validating slabcache 'filp'
--- Checked 35 slabs in 'filp'
--- Validating slabcache 'flow_cache'
--- Checked 0 slabs in 'flow_cache'
--- Validating slabcache 'fs_cache'
--- Checked 5 slabs in 'fs_cache'
--- Validating slabcache 'hugetlbfs_inode_cache'
--- Checked 1 slabs in 'hugetlbfs_inode_cache'
--- Validating slabcache 'idr_layer_cache'
--- Checked 2 slabs in 'idr_layer_cache'
--- Validating slabcache 'inet_peer_cache'
--- Checked 0 slabs in 'inet_peer_cache'
--- Validating slabcache 'inode_cache'
--- Checked 8 slabs in 'inode_cache'
--- Validating slabcache 'inotify_event_cache'
--- Checked 0 slabs in 'inotify_event_cache'
--- Validating slabcache 'inotify_watch_cache'
--- Checked 1 slabs in 'inotify_watch_cache'
--- Validating slabcache 'ip6_dst_cache'
--- Checked 1 slabs in 'ip6_dst_cache'
--- Validating slabcache 'ip_dst_cache'
--- Checked 3 slabs in 'ip_dst_cache'
--- Validating slabcache 'ip_fib_alias'
--- Checked 3 slabs in 'ip_fib_alias'
--- Validating slabcache 'ip_fib_hash'
--- Checked 3 slabs in 'ip_fib_hash'
--- Validating slabcache 'ip_mrt_cache'
--- Checked 0 slabs in 'ip_mrt_cache'
--- Validating slabcache 'isofs_inode_cache'
--- Checked 0 slabs in 'isofs_inode_cache'
--- Validating slabcache 'journal_handle'
--- Checked 0 slabs in 'journal_handle'
--- Validating slabcache 'journal_head'
--- Checked 0 slabs in 'journal_head'
--- Validating slabcache 'kiocb'
--- Checked 0 slabs in 'kiocb'
--- Validating slabcache 'kioctx'
--- Checked 0 slabs in 'kioctx'
--- Validating slabcache 'kmalloc-1024'
--- Checked 7 slabs in 'kmalloc-1024'
--- Validating slabcache 'kmalloc-128'
--- Checked 9 slabs in 'kmalloc-128'
--- Validating slabcache 'kmalloc-131072'
--- Checked 0 slabs in 'kmalloc-131072'
--- Validating slabcache 'kmalloc-16'
--- Checked 7 slabs in 'kmalloc-16'
--- Validating slabcache 'kmalloc-16384'
--- Checked 2 slabs in 'kmalloc-16384'
--- Validating slabcache 'kmalloc-192'
--- Checked 4 slabs in 'kmalloc-192'
--- Validating slabcache 'kmalloc-2048'
--- Checked 17 slabs in 'kmalloc-2048'
--- Validating slabcache 'kmalloc-256'
--- Checked 5 slabs in 'kmalloc-256'
--- Validating slabcache 'kmalloc-262144'
--- Checked 0 slabs in 'kmalloc-262144'
--- Validating slabcache 'kmalloc-32'
--- Checked 7 slabs in 'kmalloc-32'
--- Validating slabcache 'kmalloc-32768'
--- Checked 2 slabs in 'kmalloc-32768'
--- Validating slabcache 'kmalloc-4096'
--- Checked 5 slabs in 'kmalloc-4096'
--- Validating slabcache 'kmalloc-512'
--- Checked 9 slabs in 'kmalloc-512'
--- Validating slabcache 'kmalloc-64'
--- Checked 9 slabs in 'kmalloc-64'
--- Validating slabcache 'kmalloc-65536'
--- Checked 1 slabs in 'kmalloc-65536'
--- Validating slabcache 'kmalloc-8'
--- Checked 6 slabs in 'kmalloc-8'
--- Validating slabcache 'kmalloc-8192'
--- Checked 3 slabs in 'kmalloc-8192'
--- Validating slabcache 'kmalloc-96'
--- Checked 7 slabs in 'kmalloc-96'
--- Validating slabcache 'kmem_cache_node'
--- Checked 6 slabs in 'kmem_cache_node'
--- Validating slabcache 'minix_inode_cache'
--- Checked 0 slabs in 'minix_inode_cache'
--- Validating slabcache 'mm_struct'
--- Checked 6 slabs in 'mm_struct'
--- Validating slabcache 'mnt_cache'
--- Checked 3 slabs in 'mnt_cache'
--- Validating slabcache 'mqueue_inode_cache'
--- Checked 1 slabs in 'mqueue_inode_cache'
--- Validating slabcache 'names_cache'
--- Checked 0 slabs in 'names_cache'
--- Validating slabcache 'ndisc_cache'
--- Checked 1 slabs in 'ndisc_cache'
--- Validating slabcache 'nfs_direct_cache'
--- Checked 0 slabs in 'nfs_direct_cache'
--- Validating slabcache 'nfs_inode_cache'
--- Checked 0 slabs in 'nfs_inode_cache'
--- Validating slabcache 'nfs_page'
--- Checked 0 slabs in 'nfs_page'
--- Validating slabcache 'nfs_read_data'
--- Checked 1 slabs in 'nfs_read_data'
--- Validating slabcache 'nfs_write_data'
--- Checked 0 slabs in 'nfs_write_data'
--- Validating slabcache 'numa_policy'
--- Checked 0 slabs in 'numa_policy'
--- Validating slabcache 'pid'
--- Checked 8 slabs in 'pid'
--- Validating slabcache 'posix_timers_cache'
--- Checked 0 slabs in 'posix_timers_cache'
--- Validating slabcache 'proc_inode_cache'
--- Checked 26 slabs in 'proc_inode_cache'
--- Validating slabcache 'radix_tree_node'
--- Checked 5 slabs in 'radix_tree_node'
--- Validating slabcache 'reiser_inode_cache'
--- Checked 5 slabs in 'reiser_inode_cache'
--- Validating slabcache 'request_sock_TCP'
--- Checked 0 slabs in 'request_sock_TCP'
--- Validating slabcache 'request_sock_TCPv6'
--- Checked 0 slabs in 'request_sock_TCPv6'
--- Validating slabcache 'revoke_record'
--- Checked 0 slabs in 'revoke_record'
--- Validating slabcache 'revoke_table'
--- Checked 0 slabs in 'revoke_table'
--- Validating slabcache 'revokefs_inode_cache'
--- Checked 1 slabs in 'revokefs_inode_cache'
--- Validating slabcache 'rpc_buffers'
--- Checked 1 slabs in 'rpc_buffers'
--- Validating slabcache 'rpc_inode_cache'
--- Checked 0 slabs in 'rpc_inode_cache'
--- Validating slabcache 'rpc_tasks'
--- Checked 1 slabs in 'rpc_tasks'
--- Validating slabcache 'secpath_cache'
--- Checked 0 slabs in 'secpath_cache'
--- Validating slabcache 'shared_policy_node'
--- Checked 0 slabs in 'shared_policy_node'
--- Validating slabcache 'shmem_inode_cache'
--- Checked 5 slabs in 'shmem_inode_cache'
--- Validating slabcache 'sighand_cache'
--- Checked 7 slabs in 'sighand_cache'
--- Validating slabcache 'signal_cache'
--- Checked 7 slabs in 'signal_cache'
--- Validating slabcache 'sigqueue'
--- Checked 0 slabs in 'sigqueue'
--- Validating slabcache 'skbuff_fclone_cache'
--- Checked 0 slabs in 'skbuff_fclone_cache'
--- Validating slabcache 'skbuff_head_cache'
--- Checked 16 slabs in 'skbuff_head_cache'
--- Validating slabcache 'sock_inode_cache'
--- Checked 5 slabs in 'sock_inode_cache'
--- Validating slabcache 'sysfs_dir_cache'
--- Checked 7 slabs in 'sysfs_dir_cache'
--- Validating slabcache 'task_struct'
--- Checked 12 slabs in 'task_struct'
--- Validating slabcache 'tcp_bind_bucket'
--- Checked 4 slabs in 'tcp_bind_bucket'
--- Validating slabcache 'tw_sock_TCP'
--- Checked 0 slabs in 'tw_sock_TCP'
--- Validating slabcache 'tw_sock_TCPv6'
--- Checked 0 slabs in 'tw_sock_TCPv6'
--- Validating slabcache 'uid_cache'
--- Checked 2 slabs in 'uid_cache'
--- Validating slabcache 'utrace_cache'
--- Checked 0 slabs in 'utrace_cache'
--- Validating slabcache 'utrace_engine_cache'
--- Checked 0 slabs in 'utrace_engine_cache'
--- Validating slabcache 'vm_area_struct'
--- Checked 28 slabs in 'vm_area_struct'
--- Validating slabcache 'xfrm_dst_cache'
--- Checked 0 slabs in 'xfrm_dst_cache'

Re: 2.6.21-rc5-mm4 (SLUB)

[Christoph Lameter]

On Thu, 5 Apr 2007, Badari Pulavarty wrote:

> On Wed, 2007-04-04 at 21:29 -0700, Christoph Lameter wrote:
> > Here is a patch that adds validation (only for cpuslabs and partial 
> > slabs but thats where the action is). Apply this patch
> > and then do
> > 
> > echo 1 >/sys/slab/<cache-to-check>/validate
> > 
> > I suggest to boot with full debugging and then run this on the ACPI slabs.
> 
> Did this and didn't trigger any problems.

Duh. Must have been in the full slabs. Maybe I should add a tracking of 
full slabs for the debug case. Would also enable leak detection.

[PATCH] cache_k8_northbridges() overflows beyond allocation (Was: 2.6.21-rc5-mm4 (SLUB))

[Badari Pulavarty]

On Wed, 2007-04-04 at 11:04 -0700, Christoph Lameter wrote:
> On Wed, 4 Apr 2007, Badari Pulavarty wrote:
...
> 
> > *** SLUB: Freepointer corrupt in kmalloc-16@0xffff810173f172a0 Slab
> > 0xffff81017f9f8b80
> >     offset=672 flags=0x2000000000000c7 inuse=42
> > freelist=0xffff810173f172a0
> >   Bytes b4 0xffff810173f17290:  a0 72 f1 73 00 00 00 00 00 00 00 00 00
> > 00 00 00 .r\uffffs............
> >     Object 0xffff810173f172a0:  00 00 00 00 01 81 ff ff 00 00 00 00 00
> > 00 00 00 ......\uffff\uffff........
> > FreePointer 0xffff810173f172a0 -> 0xffff810100000000
> 

Found it !! After a painful capture of all the kmalloc-16 slab
allocations (400+) so far and auditing some of them, found the
culprit - who writes beyond its allocation, causing the slab
corruption.

Thanks,
Badari

cache_k8_northbridges() is storing config values to incorrect locations
(in flush_words) and also its overflowing beyond the allocation, causing
slab verification failures.

Signed-off-by: Badari Pulavarty <pbadari@us.ibm.com>
---
 arch/x86_64/kernel/k8.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Index: linux-2.6.21-rc6/arch/x86_64/kernel/k8.c
===================================================================
--- linux-2.6.21-rc6.orig/arch/x86_64/kernel/k8.c	2007-04-05 19:36:56.000000000 -0700
+++ linux-2.6.21-rc6/arch/x86_64/kernel/k8.c	2007-04-13 07:51:57.000000000 -0700
@@ -61,8 +61,8 @@ int cache_k8_northbridges(void)
 	dev = NULL;
 	i = 0;
 	while ((dev = next_k8_northbridge(dev)) != NULL) {
-		k8_northbridges[i++] = dev;
-		pci_read_config_dword(dev, 0x9c, &flush_words[i]);
+		k8_northbridges[i] = dev;
+		pci_read_config_dword(dev, 0x9c, &flush_words[i++]);
 	}
 	k8_northbridges[i] = NULL;
 	return 0;

Re: [PATCH] cache_k8_northbridges() overflows beyond allocation (Was: 2.6.21-rc5-mm4 (SLUB))

[Andi Kleen]

> 
> cache_k8_northbridges() is storing config values to incorrect locations
> (in flush_words) and also its overflowing beyond the allocation, causing
> slab verification failures.

Oops. Thanks for tracking that down, Badari.

Andrew, clear .21 candidate.

-ANdi

> 
> Signed-off-by: Badari Pulavarty <pbadari@us.ibm.com>
> ---
>  arch/x86_64/kernel/k8.c |    4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> Index: linux-2.6.21-rc6/arch/x86_64/kernel/k8.c
> ===================================================================
> --- linux-2.6.21-rc6.orig/arch/x86_64/kernel/k8.c	2007-04-05 19:36:56.000000000 -0700
> +++ linux-2.6.21-rc6/arch/x86_64/kernel/k8.c	2007-04-13 07:51:57.000000000 -0700
> @@ -61,8 +61,8 @@ int cache_k8_northbridges(void)
>  	dev = NULL;
>  	i = 0;
>  	while ((dev = next_k8_northbridge(dev)) != NULL) {
> -		k8_northbridges[i++] = dev;
> -		pci_read_config_dword(dev, 0x9c, &flush_words[i]);
> +		k8_northbridges[i] = dev;
> +		pci_read_config_dword(dev, 0x9c, &flush_words[i++]);
>  	}
>  	k8_northbridges[i] = NULL;
>  	return 0;
> 
> 
> 

Re: [PATCH] cache_k8_northbridges() overflows beyond allocation (Was: 2.6.21-rc5-mm4 (SLUB))

[Chuck Ebbert]

Andi Kleen wrote:
>> cache_k8_northbridges() is storing config values to incorrect locations
>> (in flush_words) and also its overflowing beyond the allocation, causing
>> slab verification failures.
> 
> Oops. Thanks for tracking that down, Badari.
> 
> Andrew, clear .21 candidate.
> 

2.6.20 as well. Do you want me to submit it?

Re: [PATCH] cache_k8_northbridges() overflows beyond allocation (Was: 2.6.21-rc5-mm4 (SLUB))

[Andi Kleen]

On Friday 13 April 2007 18:42:43 Chuck Ebbert wrote:
> Andi Kleen wrote:
> >> cache_k8_northbridges() is storing config values to incorrect locations
> >> (in flush_words) and also its overflowing beyond the allocation, causing
> >> slab verification failures.
> > 
> > Oops. Thanks for tracking that down, Badari.
> > 
> > Andrew, clear .21 candidate.
> > 
> 
> 2.6.20 as well. Do you want me to submit it?

After it is in .21

-Andi

Re: [PATCH] cache_k8_northbridges() overflows beyond allocation (Was: 2.6.21-rc5-mm4 (SLUB))

[Andrew Morton]

On Fri, 13 Apr 2007 17:45:37 +0200 Andi Kleen <ak@suse.de> wrote:

> 
> > 
> > cache_k8_northbridges() is storing config values to incorrect locations
> > (in flush_words) and also its overflowing beyond the allocation, causing
> > slab verification failures.
> 
> Oops. Thanks for tracking that down, Badari.
> 
> Andrew, clear .21 candidate.

OK.  And for 2.6.20.x, methinks.

Re: [PATCH] cache_k8_northbridges() overflows beyond allocation (Was: 2.6.21-rc5-mm4 (SLUB))

[Christoph Lameter]

On Fri, 13 Apr 2007, Badari Pulavarty wrote:

> On Wed, 2007-04-04 at 11:04 -0700, Christoph Lameter wrote:
> > On Wed, 4 Apr 2007, Badari Pulavarty wrote:
> ...
> > 
> > > *** SLUB: Freepointer corrupt in kmalloc-16@0xffff810173f172a0 Slab
> > > 0xffff81017f9f8b80
> > >     offset=672 flags=0x2000000000000c7 inuse=42
> > > freelist=0xffff810173f172a0
> > >   Bytes b4 0xffff810173f17290:  a0 72 f1 73 00 00 00 00 00 00 00 00 00
> > > 00 00 00 .r\uffffs............
> > >     Object 0xffff810173f172a0:  00 00 00 00 01 81 ff ff 00 00 00 00 00
> > > 00 00 00 ......\uffff\uffff........
> > > FreePointer 0xffff810173f172a0 -> 0xffff810100000000
> > 
> 
> Found it !! After a painful capture of all the kmalloc-16 slab
> allocations (400+) so far and auditing some of them, found the
> culprit - who writes beyond its allocation, causing the slab
> corruption.

Thanks. I am sorry that this was not easier for you. But as a result I 
thoroughly tested the slab corruption detection in SLUB yesterday found 
various issues and submitted  patches to Andrew that will make this 
really work well. Too late for you though.

Re: 2.6.21-rc5-mm4 (SLUB)

[Christoph Lameter]

On Wed, 4 Apr 2007, Badari Pulavarty wrote:

> free. Any ideas on how I can track down easily ? Is there
> a way to store last allocated (function, line#) and look
> around there ?

Also you may want to switch off slab merging. That will allow you to 
determine the cache involved if its not a kmalloc alloc and the slab was 
merged.

Note that switching off merging may seem to cure the problem because 
the object was corrupted after allocation and then the slab was never 
touched again. It may surface only if its merged because merging creates 
more activity on the slabs that will expose the problem.

Re: 2.6.21-rc5-mm4

[J.A. Magallón]

On Mon, 2 Apr 2007 22:47:45 -0700, Andrew Morton <akpm@linux-foundation.org> wrote:

> 
> ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/
> 
> - The oops in git-net.patch has been fixed, so that tree has been restored. 
>   It is huge.
> 
> - Added the device-mapper development tree to the -mm lineup (Alasdair
>   Kergon).  It is a quilt tree, living at
>   ftp://ftp.kernel.org/pub/linux/kernel/people/agk/patches/2.6/editing/.
> 
> - Added davidel's signalfd stuff.
> 

Something strange happens to me with this kernel.
Gnome Screensaver hangs the box. Locked solid, no atl-sysrq-raw,
no ssh. Nada.
Do not know if it's itself, or something in X dpms.

Somebody else ?

--
J.A. Magallon <jamagallon()ono!com>     \               Software is like sex:
                                         \         It's better when it's free
Mandriva Linux release 2007.1 (Cooker) for i586
Linux 2.6.20-jam08 (gcc 4.1.2 20070302 (prerelease) (4.1.2-1mdv2007.1)) #1 SMP PREEMPT

Re: 2.6.21-rc5-mm4

[Andrew Morton]

On Wed, 4 Apr 2007 00:40:05 +0200
"J.A. Magall__n" <jamagallon@ono.com> wrote:

> On Mon, 2 Apr 2007 22:47:45 -0700, Andrew Morton <akpm@linux-foundation.org> wrote:
> 
> > 
> > ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/
> > 
> > - The oops in git-net.patch has been fixed, so that tree has been restored. 
> >   It is huge.
> > 
> > - Added the device-mapper development tree to the -mm lineup (Alasdair
> >   Kergon).  It is a quilt tree, living at
> >   ftp://ftp.kernel.org/pub/linux/kernel/people/agk/patches/2.6/editing/.
> > 
> > - Added davidel's signalfd stuff.
> > 
> 
> Something strange happens to me with this kernel.
> Gnome Screensaver hangs the box. Locked solid, no atl-sysrq-raw,
> no ssh. Nada.
> Do not know if it's itself, or something in X dpms.
> 
> Somebody else ?
> 

Do you know what the particualr screensaver is trying to do?  Does it do
whizzy 3d graphics, or does it just blank the screen, or...  ?

Re: 2.6.21-rc5-mm4

[J.A. Magallón]

On Tue, 3 Apr 2007 15:51:35 -0700, Andrew Morton <akpm@linux-foundation.org> wrote:

> On Wed, 4 Apr 2007 00:40:05 +0200
> "J.A. Magall__n" <jamagallon@ono.com> wrote:
> 
> > On Mon, 2 Apr 2007 22:47:45 -0700, Andrew Morton <akpm@linux-foundation.org> wrote:
> > 
> > > 
> > > ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/
> > > 
> > > - The oops in git-net.patch has been fixed, so that tree has been restored. 
> > >   It is huge.
> > > 
> > > - Added the device-mapper development tree to the -mm lineup (Alasdair
> > >   Kergon).  It is a quilt tree, living at
> > >   ftp://ftp.kernel.org/pub/linux/kernel/people/agk/patches/2.6/editing/.
> > > 
> > > - Added davidel's signalfd stuff.
> > > 
> > 
> > Something strange happens to me with this kernel.
> > Gnome Screensaver hangs the box. Locked solid, no atl-sysrq-raw,
> > no ssh. Nada.
> > Do not know if it's itself, or something in X dpms.
> > 
> > Somebody else ?
> > 
> 
> Do you know what the particualr screensaver is trying to do?  Does it do
> whizzy 3d graphics, or does it just blank the screen, or...  ?

Just the 'Floating Feet', I love it ;).... (just 2D, not linked to any
libGL...).
Anyways, I have just remembered I use the (in)famous nVidia driver.
Will try to reproduce without it. This was more like a probe to see if
somebody else is suffering it...

--
J.A. Magallon <jamagallon()ono!com>     \               Software is like sex:
                                         \         It's better when it's free
Mandriva Linux release 2007.1 (Cooker) for i586
Linux 2.6.20-jam08 (gcc 4.1.2 20070302 (prerelease) (4.1.2-1mdv2007.1)) #1 SMP PREEMPT

Re: 2.6.21-rc5-mm4

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 449 bytes --]

On Wed, 04 Apr 2007 00:58:26 +0200, "J.A. =?UTF-8?B?TWFnYWxsw7Nu?=" said:

> Anyways, I have just remembered I use the (in)famous nVidia driver.
> Will try to reproduce without it. This was more like a probe to see if
> somebody else is suffering it...

The nVidia driver will get some truly astounding indigestion if you accidentally
upgrade your xorg userspace libraries and forget to re-install the nVidia
userspace.  Is that what's biting you?


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: 2.6.21-rc5-mm4

[J.A. Magallón]

On Tue, 03 Apr 2007 19:22:47 -0400, Valdis.Kletnieks@vt.edu wrote:

> On Wed, 04 Apr 2007 00:58:26 +0200, "J.A. =?UTF-8?B?TWFnYWxsw7Nu?=" said:
> 
> > Anyways, I have just remembered I use the (in)famous nVidia driver.
> > Will try to reproduce without it. This was more like a probe to see if
> > somebody else is suffering it...
> 
> The nVidia driver will get some truly astounding indigestion if you accidentally
> upgrade your xorg userspace libraries and forget to re-install the nVidia
> userspace.  Is that what's biting you?
> 

No, in my distro (mandriva), libraries and drivers from ndivia are in different
places so they don't get overriden in xorg reinstall. Use is controlled
with ld.so.conf and ModulePath.
In fact, X works fine until the saver plays on...

--
J.A. Magallon <jamagallon()ono!com>     \               Software is like sex:
                                         \         It's better when it's free
Mandriva Linux release 2007.1 (Cooker) for i586
Linux 2.6.20-jam08 (gcc 4.1.2 20070302 (prerelease) (4.1.2-1mdv2007.1)) #1 SMP PREEMPT

Re: 2.6.21-rc5-mm4

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1270 bytes --]

On Mon, 02 Apr 2007 22:47:45 PDT, Andrew Morton said:
> ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/

So I was looking at a patch for ACPI_SLEEP that went around a moment ago,
and ended up doing a 'make menuconfig'.

I do a '/ACPI_SLEEP' inside that, and I get this output:

x Symbol: ACPI_SLEEP [=n]                                                 x  
  x Prompt: Sleep States                                                    x  
  x   Defined at drivers/acpi/Kconfig:49                                    x  
  x   Depends on: !X86_NUMAQ && !X86_VISWS && !IA64_HP_SIM && (IA64 || X86) x  
  x   Location:                                                             x  
  x     -> Power management options                                         x  
  x       -> Power Management support (PM [=y])                             x  
  x         -> ACPI (Advanced Configuration and Power Interface) Support    x  
  x           -> ACPI Support (ACPI [=y])   

Only problem is that drivers/acpi/Kconfig:49 says:

config ACPI_SLEEP
        bool "Sleep States"
        depends on X86 && (!SMP || SUSPEND_SMP)
        depends on PM
        default y

Can somebody explain why I'm seeing the different Depends on in the two places?

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: 2.6.21-rc5-mm4

[Randy Dunlap]

On Tue, 03 Apr 2007 21:35:54 -0400 Valdis.Kletnieks@vt.edu wrote:

> On Mon, 02 Apr 2007 22:47:45 PDT, Andrew Morton said:
> > ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/
> 
> So I was looking at a patch for ACPI_SLEEP that went around a moment ago,
> and ended up doing a 'make menuconfig'.
> 
> I do a '/ACPI_SLEEP' inside that, and I get this output:
> 
> x Symbol: ACPI_SLEEP [=n]                                                 x  
>   x Prompt: Sleep States                                                    x  
>   x   Defined at drivers/acpi/Kconfig:49                                    x  
>   x   Depends on: !X86_NUMAQ && !X86_VISWS && !IA64_HP_SIM && (IA64 || X86) x  
>   x   Location:                                                             x  
>   x     -> Power management options                                         x  
>   x       -> Power Management support (PM [=y])                             x  
>   x         -> ACPI (Advanced Configuration and Power Interface) Support    x  
>   x           -> ACPI Support (ACPI [=y])   
> 
> Only problem is that drivers/acpi/Kconfig:49 says:
> 
> config ACPI_SLEEP
>         bool "Sleep States"
>         depends on X86 && (!SMP || SUSPEND_SMP)
>         depends on PM
>         default y
> 
> Can somebody explain why I'm seeing the different Depends on in the two places?

Due to the top of drivers/acpi/Kconfig:

menu "ACPI (Advanced Configuration and Power Interface) Support"
	depends on !X86_NUMAQ
	depends on !X86_VISWS
	depends on !IA64_HP_SIM
	depends on IA64 || X86
	depends on PM


---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

Re: 2.6.21-rc5-mm4

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1160 bytes --]

On Tue, 03 Apr 2007 19:20:04 PDT, Randy Dunlap said:
> On Tue, 03 Apr 2007 21:35:54 -0400 Valdis.Kletnieks@vt.edu wrote:
> > I do a '/ACPI_SLEEP' inside that, and I get this output:
> > 
> > x Symbol: ACPI_SLEEP [=n]                                                 x
> >   x   Depends on: !X86_NUMAQ && !X86_VISWS && !IA64_HP_SIM && (IA64 || X86)

> > Only problem is that drivers/acpi/Kconfig:49 says:
> > 
> > config ACPI_SLEEP
> >         bool "Sleep States"
> >         depends on X86 && (!SMP || SUSPEND_SMP)
> >         depends on PM
> >         default y
> > 
> > Can somebody explain why I'm seeing the different Depends on in the two places?
> 
> Due to the top of drivers/acpi/Kconfig:
> 
> menu "ACPI (Advanced Configuration and Power Interface) Support"
> 	depends on !X86_NUMAQ
> 	depends on !X86_VISWS
> 	depends on !IA64_HP_SIM
> 	depends on IA64 || X86
> 	depends on PM

Ah. Further investigation shows that the "&& (!SMP || SUSPEND_SMP)" part
failed to show up because the line was too long - running 'make menuconfig'
in an 140-col or so xterm made them show up.

Anybody ambitious enough to add a "line too long" indicator of some sort? :)


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: 2.6.21-rc5-mm4

[Randy Dunlap]

Valdis.Kletnieks@vt.edu wrote:
> On Tue, 03 Apr 2007 19:20:04 PDT, Randy Dunlap said:
>> On Tue, 03 Apr 2007 21:35:54 -0400 Valdis.Kletnieks@vt.edu wrote:
>>> I do a '/ACPI_SLEEP' inside that, and I get this output:
>>>
>>> x Symbol: ACPI_SLEEP [=n]                                                 x
>>>   x   Depends on: !X86_NUMAQ && !X86_VISWS && !IA64_HP_SIM && (IA64 || X86)
> 
>>> Only problem is that drivers/acpi/Kconfig:49 says:
>>>
>>> config ACPI_SLEEP
>>>         bool "Sleep States"
>>>         depends on X86 && (!SMP || SUSPEND_SMP)
>>>         depends on PM
>>>         default y
>>>
>>> Can somebody explain why I'm seeing the different Depends on in the two places?
>> Due to the top of drivers/acpi/Kconfig:
>>
>> menu "ACPI (Advanced Configuration and Power Interface) Support"
>> 	depends on !X86_NUMAQ
>> 	depends on !X86_VISWS
>> 	depends on !IA64_HP_SIM
>> 	depends on IA64 || X86
>> 	depends on PM
> 
> Ah. Further investigation shows that the "&& (!SMP || SUSPEND_SMP)" part
> failed to show up because the line was too long - running 'make menuconfig'
> in an 140-col or so xterm made them show up.
> 
> Anybody ambitious enough to add a "line too long" indicator of some sort? :)
> 

Good luck.  But the symbols are there.  Just use left/right arrow keys
to scroll the display left/right and you can see them.  Now if you just
had that indicator to tell you that you Need to scroll to see more text...

-- 
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

Re: 2.6.21-rc5-mm4

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 973 bytes --]

On Tue, 03 Apr 2007 20:37:42 PDT, Randy Dunlap said:
>
> Good luck.  But the symbols are there.  Just use left/right arrow keys
> to scroll the display left/right and you can see them.  Now if you just
> had that indicator to tell you that you Need to scroll to see more text...

Exactly. :)  I had the incredible bad luck that the line got cut off at the
end of a CONFIG_ symbol that made sense - if it had showed up *half* a symbol,
I'd have gone investigating. ;) (Even a '>' or '<' saying data offscreen to
right or left would be sufficient, if somebody wants a small but productive
kernel (config system actually) task to hack on.)

I'd code it myself, but I have an SL8500 to install, and need to figure out
how my laptop made it into the bag this morning still up and running (I hit
the power button, it seemed to power down - blank screen, power light off,
but syslog msgs prove it was up and running for another 4 hours before it
shut down on a thermal check...)


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: 2.6.21-rc5-mm4

[Sam Ravnborg]

On Wed, Apr 04, 2007 at 01:55:08PM -0400, Valdis.Kletnieks@vt.edu wrote:
> On Tue, 03 Apr 2007 20:37:42 PDT, Randy Dunlap said:
> >
> > Good luck.  But the symbols are there.  Just use left/right arrow keys
> > to scroll the display left/right and you can see them.  Now if you just
> > had that indicator to tell you that you Need to scroll to see more text...
> 
> Exactly. :)  I had the incredible bad luck that the line got cut off at the
> end of a CONFIG_ symbol that made sense - if it had showed up *half* a symbol,
> I'd have gone investigating. ;) (Even a '>' or '<' saying data offscreen to
> right or left would be sufficient, if somebody wants a small but productive
> kernel (config system actually) task to hack on.)
> 
> I'd code it myself, but I have an SL8500 to install, and need to figure out
> how my laptop made it into the bag this morning still up and running (I hit
> the power button, it seemed to power down - blank screen, power light off,
> but syslog msgs prove it was up and running for another 4 hours before it
> shut down on a thermal check...)

If you do not find time to do it try to ping me in a week or so.
Should be trivial to do but away from my dev box atm.

   Sam

[-mm patch] net/: possible cleanups

[Adrian Bunk]

On Mon, Apr 02, 2007 at 10:47:45PM -0700, Andrew Morton wrote:
>...
> Changes since 2.6.21-rc5-mm3:
>...
>  git-net.patch
>...
>  git trees.
>...


This patch contains the following possible cleanups:
- make the following needlessly global variables static:
  - core/rtnetlink.c: struct rtnl_msg_handlers[]
  - netfilter/nf_conntrack_proto.c: struct nf_ct_protos[] 
- make the following needlessly global functions static:
  - core/rtnetlink.c: rtnl_dump_all()
  - netlink/af_netlink.c: netlink_queue_skip()

Signed-off-by: Adrian Bunk <bunk@stusta.de>

---

 include/net/netfilter/nf_conntrack_l4proto.h |    1 -
 include/net/netlink.h                        |    2 --
 include/net/rtnetlink.h                      |    1 -
 net/core/rtnetlink.c                         |    6 ++----
 net/netfilter/nf_conntrack_proto.c           |    2 +-
 net/netlink/af_netlink.c                     |    4 ++--
 6 files changed, 5 insertions(+), 11 deletions(-)

--- linux-2.6.21-rc5-mm4/include/net/rtnetlink.h.old	2007-04-03 23:58:30.000000000 +0200
+++ linux-2.6.21-rc5-mm4/include/net/rtnetlink.h	2007-04-03 23:58:35.000000000 +0200
@@ -13,7 +13,6 @@
 			      rtnl_doit_func, rtnl_dumpit_func);
 extern int	rtnl_unregister(int protocol, int msgtype);
 extern void	rtnl_unregister_all(int protocol);
-extern int	rtnl_dump_all(struct sk_buff *skb, struct netlink_callback *cb);
 
 static inline int rtnl_msg_family(struct nlmsghdr *nlh)
 {
--- linux-2.6.21-rc5-mm4/net/core/rtnetlink.c.old	2007-04-03 23:34:49.000000000 +0200
+++ linux-2.6.21-rc5-mm4/net/core/rtnetlink.c	2007-04-03 23:58:44.000000000 +0200
@@ -101,7 +101,7 @@
 	return 0;
 }
 
-struct rtnl_link *rtnl_msg_handlers[NPROTO];
+static struct rtnl_link *rtnl_msg_handlers[NPROTO];
 
 static inline int rtm_msgindex(int msgtype)
 {
@@ -771,7 +771,7 @@
 	return err;
 }
 
-int rtnl_dump_all(struct sk_buff *skb, struct netlink_callback *cb)
+static int rtnl_dump_all(struct sk_buff *skb, struct netlink_callback *cb)
 {
 	int idx;
 	int s_idx = cb->family;
@@ -795,8 +795,6 @@
 	return skb->len;
 }
 
-EXPORT_SYMBOL_GPL(rtnl_dump_all);
-
 void rtmsg_ifinfo(int type, struct net_device *dev, unsigned change)
 {
 	struct sk_buff *skb;
--- linux-2.6.21-rc5-mm4/include/net/netfilter/nf_conntrack_l4proto.h.old	2007-04-04 01:22:41.000000000 +0200
+++ linux-2.6.21-rc5-mm4/include/net/netfilter/nf_conntrack_l4proto.h	2007-04-04 01:22:50.000000000 +0200
@@ -97,7 +97,6 @@
 extern struct nf_conntrack_l4proto nf_conntrack_l4proto_generic;
 
 #define MAX_NF_CT_PROTO 256
-extern struct nf_conntrack_l4proto **nf_ct_protos[PF_MAX];
 
 extern struct nf_conntrack_l4proto *
 __nf_ct_l4proto_find(u_int16_t l3proto, u_int8_t l4proto);
--- linux-2.6.21-rc5-mm4/net/netfilter/nf_conntrack_proto.c.old	2007-04-04 00:00:26.000000000 +0200
+++ linux-2.6.21-rc5-mm4/net/netfilter/nf_conntrack_proto.c	2007-04-04 00:00:43.000000000 +0200
@@ -28,7 +28,7 @@
 #include <net/netfilter/nf_conntrack_l4proto.h>
 #include <net/netfilter/nf_conntrack_core.h>
 
-struct nf_conntrack_l4proto **nf_ct_protos[PF_MAX] __read_mostly;
+static struct nf_conntrack_l4proto **nf_ct_protos[PF_MAX] __read_mostly;
 struct nf_conntrack_l3proto *nf_ct_l3protos[AF_MAX] __read_mostly;
 EXPORT_SYMBOL_GPL(nf_ct_l3protos);
 
--- linux-2.6.21-rc5-mm4/include/net/netlink.h.old	2007-04-04 00:01:21.000000000 +0200
+++ linux-2.6.21-rc5-mm4/include/net/netlink.h	2007-04-04 00:01:52.000000000 +0200
@@ -217,8 +217,6 @@
 extern void		netlink_run_queue(struct sock *sk, unsigned int *qlen,
 					  int (*cb)(struct sk_buff *,
 						    struct nlmsghdr *));
-extern void		netlink_queue_skip(struct nlmsghdr *nlh,
-					   struct sk_buff *skb);
 extern int		nlmsg_notify(struct sock *sk, struct sk_buff *skb,
 				     u32 pid, unsigned int group, int report,
 				     gfp_t flags);
--- linux-2.6.21-rc5-mm4/net/netlink/af_netlink.c.old	2007-04-04 00:01:59.000000000 +0200
+++ linux-2.6.21-rc5-mm4/net/netlink/af_netlink.c	2007-04-04 00:02:22.000000000 +0200
@@ -117,6 +117,7 @@
 
 static int netlink_dump(struct sock *sk);
 static void netlink_destroy_callback(struct netlink_callback *cb);
+static void netlink_queue_skip(struct nlmsghdr *nlh, struct sk_buff *skb);
 
 static DEFINE_RWLOCK(nl_table_lock);
 static atomic_t nl_table_users = ATOMIC_INIT(0);
@@ -1555,7 +1556,7 @@
  * Pulls the given netlink message off the socket buffer so the next
  * call to netlink_queue_run() will not reconsider the message.
  */
-void netlink_queue_skip(struct nlmsghdr *nlh, struct sk_buff *skb)
+static void netlink_queue_skip(struct nlmsghdr *nlh, struct sk_buff *skb)
 {
 	int msglen = NLMSG_ALIGN(nlh->nlmsg_len);
 
@@ -1838,7 +1839,6 @@
 
 EXPORT_SYMBOL(netlink_ack);
 EXPORT_SYMBOL(netlink_run_queue);
-EXPORT_SYMBOL(netlink_queue_skip);
 EXPORT_SYMBOL(netlink_broadcast);
 EXPORT_SYMBOL(netlink_dump_start);
 EXPORT_SYMBOL(netlink_kernel_create);

[-mm patch] unexport pci_proc_attach_device

[Adrian Bunk]

On Mon, Apr 02, 2007 at 10:47:45PM -0700, Andrew Morton wrote:
>...
> Changes since 2.6.21-rc5-mm3:
>...
> +fix-82875-pci-setup.patch
>...
>  Misc
>...


pci_proc_attach_device() no longer has any modular user.

Signed-off-by: Adrian Bunk <bunk@stusta.de>

---
--- linux-2.6.21-rc5-mm4/drivers/pci/proc.c.old	2007-04-04 17:06:04.000000000 +0200
+++ linux-2.6.21-rc5-mm4/drivers/pci/proc.c	2007-04-04 17:08:35.000000000 +0200
@@ -480,7 +480,6 @@
 __initcall(pci_proc_init);
 
 #ifdef CONFIG_HOTPLUG
-EXPORT_SYMBOL(pci_proc_attach_device);
 EXPORT_SYMBOL(pci_proc_detach_bus);
 #endif
 

Re: 2.6.21-rc5-mm4

[Antonino A. Daplas]

On Mon, 2007-04-02 at 22:47 -0700, Andrew Morton wrote:
> ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/
> 
> - The oops in git-net.patch has been fixed, so that tree has been restored. 
>   It is huge.
> 
> - Added the device-mapper development tree to the -mm lineup (Alasdair
>   Kergon).  It is a quilt tree, living at
>   ftp://ftp.kernel.org/pub/linux/kernel/people/agk/patches/2.6/editing/.
> 
> - Added davidel's signalfd stuff.
> 
> 
> 

I'm getting a kernel panic intermittently, approximately 50% of boots.
The tracing is not always the same, but it always dies on an
atomic_bitop operation.  Here are two hand-copied tracings (for the life
of me, I can't make netconsole work).


/---------------------------First Tracing----------------------/
Oops: 0000 [#1]
last sysfs file: class/firmware/microcode
Modules linked in: ...
....
...
CPU: 0
EIP: ...
EFLAGS:...
EIP is at find_next_zero_bit
...
...
...
Process set_disk_settin
Call Trace:
show_trace_log
show_stack_log
show_register
die
do_page_fault
error_code
recalc_task_prio
activate_task
try_to_wake_up
deault_wake_function
__wake_up_common
__wake_up
sock_def_readable
soc_queue_rev_skb
udp_queue_rcv_skb
__udp4_libr_rcv
udp_rcv
ip_local_delivery
ip_rcv
netif_receive_skb
rtl8139_poll
net_rx_action
__do_soft_irq
do_softirq
irq_exit
do_IRQ
common_interrupt

/---------------------- Second Tracing --------------------------/
CPU: 0
EIP: ...
EFLAGS:...
EIP is at find_next_zero_bit
...
...
...
Process sshd
Call Trace:
show_trace_log
show_stack_log
show_register
die
do_page_fault
error_code
recalc_task_prio
enqueue_task
activate_task
try_to_wake_up
wake_up_state
signal_wake_up
__group_complete_signal
__group_send_signal
group_send_sig_info
send_group_sig_info
it_real_fn
run_hrtimer_softirq
__do_softirq
irq_exit
smp_apic_timer_interrupt
apic_timer_interrupt
error_code

EIP: [<c01bfa1c>. find_next_zero_bit+...

Tony

PS: I might try use a serial console and bisection, but this might take
me a few days.

Re: 2.6.21-rc5-mm4

[Andrew Morton]

On Thu, 05 Apr 2007 05:56:35 +0800
"Antonino A. Daplas" <adaplas@gmail.com> wrote:

> On Mon, 2007-04-02 at 22:47 -0700, Andrew Morton wrote:
> > ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/
> > 
> > - The oops in git-net.patch has been fixed, so that tree has been restored. 
> >   It is huge.
> > 
> > - Added the device-mapper development tree to the -mm lineup (Alasdair
> >   Kergon).  It is a quilt tree, living at
> >   ftp://ftp.kernel.org/pub/linux/kernel/people/agk/patches/2.6/editing/.
> > 
> > - Added davidel's signalfd stuff.
> > 
> > 
> > 
> 
> I'm getting a kernel panic intermittently, approximately 50% of boots.
> The tracing is not always the same, but it always dies on an
> atomic_bitop operation.  Here are two hand-copied tracings (for the life
> of me, I can't make netconsole work).
> 
> 
> /---------------------------First Tracing----------------------/
> Oops: 0000 [#1]
> last sysfs file: class/firmware/microcode
> Modules linked in: ...
> ....
> ...
> CPU: 0
> EIP: ...
> EFLAGS:...
> EIP is at find_next_zero_bit
> ...
> ...
> ...
> Process set_disk_settin
> Call Trace:
> show_trace_log
> show_stack_log
> show_register
> die
> do_page_fault
> error_code
> recalc_task_prio
> activate_task
> try_to_wake_up
> deault_wake_function
> __wake_up_common
> __wake_up
> sock_def_readable
> soc_queue_rev_skb
> udp_queue_rcv_skb
> __udp4_libr_rcv
> udp_rcv
> ip_local_delivery
> ip_rcv
> netif_receive_skb
> rtl8139_poll
> net_rx_action
> __do_soft_irq
> do_softirq
> irq_exit
> do_IRQ
> common_interrupt

Thanks - that'll be the CPU scheduler changes.

Con has produced a patch or two which might address this but afaik we don't
yet have a definitive fix?

I believe that reverting
sched-implement-staircase-deadline-cpu-scheduler-staircase-improvements.patch
will prevent it.

Re: 2.6.21-rc5-mm4

[Con Kolivas]

On Thursday 05 April 2007 08:10, Andrew Morton wrote:
> Thanks - that'll be the CPU scheduler changes.
>
> Con has produced a patch or two which might address this but afaik we don't
> yet have a definitive fix?
>
> I believe that reverting
> sched-implement-staircase-deadline-cpu-scheduler-staircase-improvements.pat
>ch will prevent it.

I posted a definitive fix which Michal tested for me offlist. Subject was:
 [PATCH] sched: implement staircase deadline cpu scheduler improvements fix

Sorry about relative noise prior to that. Akpm please pick it up.

Here again just in case.

---
Use of memset was bogus. Fix it.

Fix exiting recalc_task_prio without p->array being updated.

Microoptimisation courtesy of Dmitry Adamushko <dmitry.adamushko@gmail.com>

Signed-off-by: Con Kolivas <kernel@kolivas.org>

---
 kernel/sched.c |   17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)

Index: linux-2.6.21-rc5-mm4/kernel/sched.c
===================================================================
--- linux-2.6.21-rc5-mm4.orig/kernel/sched.c	2007-04-04 12:14:29.000000000 +1000
+++ linux-2.6.21-rc5-mm4/kernel/sched.c	2007-04-04 12:49:39.000000000 +1000
@@ -683,11 +683,13 @@ static void dequeue_task(struct task_str
  * The task is being queued on a fresh array so it has its entitlement
  * bitmap cleared.
  */
-static inline void task_new_array(struct task_struct *p, struct rq *rq)
+static void task_new_array(struct task_struct *p, struct rq *rq,
+			   struct prio_array *array)
 {
 	bitmap_zero(p->bitmap, PRIO_RANGE);
 	p->rotation = rq->prio_rotation;
 	p->time_slice = p->quota;
+	p->array = array;
 }
 
 /* Find the first slot from the relevant prio_matrix entry */
@@ -709,6 +711,8 @@ static inline int next_entitled_slot(str
 	DECLARE_BITMAP(tmp, PRIO_RANGE);
 	int search_prio, uprio = USER_PRIO(p->static_prio);
 
+	if (!rq->prio_level[uprio])
+		rq->prio_level[uprio] = MAX_RT_PRIO;
 	/*
 	 * Only priorities equal to the prio_level and above for their
 	 * static_prio are acceptable, and only if it's not better than
@@ -736,11 +740,8 @@ static inline int next_entitled_slot(str
 
 static void queue_expired(struct task_struct *p, struct rq *rq)
 {
-	p->array = rq->expired;
-	task_new_array(p, rq);
+	task_new_array(p, rq, rq->expired);
 	p->prio = p->normal_prio = first_prio_slot(p);
-	p->time_slice = p->quota;
-	p->rotation = rq->prio_rotation;
 }
 
 #ifdef CONFIG_SMP
@@ -800,9 +801,9 @@ static void recalc_task_prio(struct task
 			queue_expired(p, rq);
 			return;
 		} else
-			task_new_array(p, rq);
+			task_new_array(p, rq, array);
 	} else
-		task_new_array(p, rq);
+		task_new_array(p, rq, array);
 
 	queue_prio = next_entitled_slot(p, rq);
 	if (queue_prio >= MAX_PRIO) {
@@ -3445,7 +3446,7 @@ EXPORT_SYMBOL(sub_preempt_count);
 
 static inline void reset_prio_levels(struct rq *rq)
 {
-	memset(rq->prio_level, MAX_RT_PRIO, ARRAY_SIZE(rq->prio_level));
+	memset(rq->prio_level, 0, sizeof(int) * PRIO_RANGE);
 }
 
 /*

-- 
-ck

Re: 2.6.21-rc5-mm4

[Antonino A. Daplas]

On Thu, 2007-04-05 at 08:38 +1000, Con Kolivas wrote:
> On Thursday 05 April 2007 08:10, Andrew Morton wrote:
> > Thanks - that'll be the CPU scheduler changes.
> >
> > Con has produced a patch or two which might address this but afaik we don't
> > yet have a definitive fix?
> >
> > I believe that reverting
> > sched-implement-staircase-deadline-cpu-scheduler-staircase-improvements.pat
> >ch will prevent it.
> 
> I posted a definitive fix which Michal tested for me offlist. Subject was:
>  [PATCH] sched: implement staircase deadline cpu scheduler improvements fix
> 
> Sorry about relative noise prior to that. Akpm please pick it up.
> 
> Here again just in case.
> 

Rebooted a few times, I can confirm that this patch fixes this.

Thanks

Tony

Re: 2.6.21-rc5-mm4

[Antonino A. Daplas]

On Mon, 2007-04-02 at 22:47 -0700, Andrew Morton wrote:
> ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/
> 
> - The oops in git-net.patch has been fixed, so that tree has been restored. 
>   It is huge.
> 
> - Added the device-mapper development tree to the -mm lineup (Alasdair
>   Kergon).  It is a quilt tree, living at
>   ftp://ftp.kernel.org/pub/linux/kernel/people/agk/patches/2.6/editing/.
> 
> - Added davidel's signalfd stuff.
> 
> 
> 

I see this tracing (from the lock-dependency validator?) for several -mm
versions.  This is from a Silan ethernet card (CONFIG_SC92031).

00:0b.0 Ethernet controller: Hangzhou Silan Microelectronics Co., Ltd.
Unknown device 2031 (rev 01)

Other than the tracing, I'm not having any problems.

Tony

======================================================
[ INFO: soft-safe -> soft-unsafe lock order detected ]
2.6.21-rc5-mm4-default #44
------------------------------------------------------
ip/3036 [HC0[0]:SC0[2]:HE1:SE0] is trying to acquire:
 (&priv->lock){--..}, at: [<d886ad22>] sc92031_set_multicast_list
+0x14/0x2d [sc92031]

and this task is already holding:
 (&dev->_xmit_lock){-...}, at: [<c02568b3>] dev_mc_upload+0x14/0x3a
which would create a new lock dependency:
 (&dev->_xmit_lock){-...} -> (&priv->lock){--..}

but this new dependency connects a soft-irq-safe lock:
 (&mc->mca_lock){-+..}
... which became soft-irq-safe at:
  [<c012f5b5>] __lock_acquire+0x3d7/0xb93
  [<c012fdd9>] lock_acquire+0x68/0x82
  [<c02aaf2a>] _spin_lock_bh+0x30/0x3d
  [<d8c53833>] mld_ifc_timer_expire+0x15b/0x21d [ipv6]
  [<c011cbd6>] run_timer_softirq+0xf1/0x14e
  [<c011a5a6>] __do_softirq+0x46/0x9c
  [<c011a629>] do_softirq+0x2d/0x46
  [<c011a9d5>] irq_exit+0x3b/0x6b
  [<c01057f0>] do_IRQ+0x5e/0x76
  [<c0103f16>] common_interrupt+0x2e/0x34
  [<c02ab3b9>] error_code+0x71/0x78
  [<ffffffff>] 0xffffffff

to a soft-irq-unsafe lock:
 (&priv->lock){--..}
... which became soft-irq-unsafe at:
...  [<c012f649>] __lock_acquire+0x46b/0xb93
  [<c012fdd9>] lock_acquire+0x68/0x82
  [<c02aaeed>] _spin_lock+0x2b/0x38
  [<d886b729>] sc92031_open+0xcc/0x16f [sc92031]
  [<c02549e7>] dev_open+0x33/0x6e
  [<c025337f>] dev_change_flags+0x57/0x10b
  [<c028b237>] devinet_ioctl+0x235/0x546
  [<c028b837>] inet_ioctl+0x89/0xaa
  [<c0249cd1>] sock_ioctl+0x1ac/0x1ca
  [<c0170a70>] do_ioctl+0x1c/0x53
  [<c0170c93>] vfs_ioctl+0x1ec/0x203
  [<c0170cf3>] sys_ioctl+0x49/0x62
  [<c0103ca8>] sysenter_past_esp+0x5d/0x99
  [<ffffffff>] 0xffffffff

other info that might help us debug this:

2 locks held by ip/3036:
 #0:  (rtnl_mutex){--..}, at: [<c02a9ea5>] mutex_lock+0x24/0x28
 #1:  (&dev->_xmit_lock){-...}, at: [<c02568b3>] dev_mc_upload+0x14/0x3a

the soft-irq-safe lock's dependencies:
-> (&mc->mca_lock){-+..} ops: 9 {
   initial-use  at:
                        [<c012f664>] __lock_acquire+0x486/0xb93
                        [<c012fdd9>] lock_acquire+0x68/0x82
                        [<c02aaf2a>] _spin_lock_bh+0x30/0x3d
                        [<d8c53983>] igmp6_group_added+0x1b/0x120 [ipv6]
                        [<d8c54175>] ipv6_dev_mc_inc+0x2f9/0x346 [ipv6]
                        [<d8c42f7f>] ipv6_add_dev+0x232/0x240 [ipv6]
                        [<d8c1430b>] ____versions+0x1e8b/0xfffff9c8
[x_tables]
                        [<d8c141d4>] ____versions+0x1d54/0xfffff9c8
[x_tables]
                        [<c01365eb>] sys_init_module+0x1252/0x138f
                        [<c0103ca8>] sysenter_past_esp+0x5d/0x99
                        [<ffffffff>] 0xffffffff
   in-softirq-W at:
                        [<c012f5b5>] __lock_acquire+0x3d7/0xb93
                        [<c012fdd9>] lock_acquire+0x68/0x82
                        [<c02aaf2a>] _spin_lock_bh+0x30/0x3d
                        [<d8c53833>] mld_ifc_timer_expire+0x15b/0x21d
[ipv6]
                        [<c011cbd6>] run_timer_softirq+0xf1/0x14e
                        [<c011a5a6>] __do_softirq+0x46/0x9c
                        [<c011a629>] do_softirq+0x2d/0x46
                        [<c011a9d5>] irq_exit+0x3b/0x6b
                        [<c01057f0>] do_IRQ+0x5e/0x76
                        [<c0103f16>] common_interrupt+0x2e/0x34
                        [<c02ab3b9>] error_code+0x71/0x78
                        [<ffffffff>] 0xffffffff
   hardirq-on-W at:
                        [<c012f61f>] __lock_acquire+0x441/0xb93
                        [<c012fdd9>] lock_acquire+0x68/0x82
                        [<c02aaf2a>] _spin_lock_bh+0x30/0x3d
                        [<d8c53983>] igmp6_group_added+0x1b/0x120 [ipv6]
                        [<d8c54175>] ipv6_dev_mc_inc+0x2f9/0x346 [ipv6]
                        [<d8c42f7f>] ipv6_add_dev+0x232/0x240 [ipv6]
                        [<d8c1430b>] ____versions+0x1e8b/0xfffff9c8
[x_tables]
                        [<d8c141d4>] ____versions+0x1d54/0xfffff9c8
[x_tables]
                        [<c01365eb>] sys_init_module+0x1252/0x138f
                        [<c0103ca8>] sysenter_past_esp+0x5d/0x99
                        [<ffffffff>] 0xffffffff
 }
 ... key      at: [<d8c75bbc>] __key.29988+0x0/0xfffe9535 [ipv6]
 -> (&dev->_xmit_lock){-...} ops: 18 {
    initial-use  at:
                          [<c012f664>] __lock_acquire+0x486/0xb93
                          [<c012fdd9>] lock_acquire+0x68/0x82
                          [<c02aaf2a>] _spin_lock_bh+0x30/0x3d
                          [<c02568b3>] dev_mc_upload+0x14/0x3a
                          [<c0253359>] dev_change_flags+0x31/0x10b
                          [<c028b237>] devinet_ioctl+0x235/0x546
                          [<c028b837>] inet_ioctl+0x89/0xaa
                          [<c0249cd1>] sock_ioctl+0x1ac/0x1ca
                          [<c0170a70>] do_ioctl+0x1c/0x53
                          [<c0170c93>] vfs_ioctl+0x1ec/0x203
                          [<c0170cf3>] sys_ioctl+0x49/0x62
                          [<c0103ca8>] sysenter_past_esp+0x5d/0x99
                          [<ffffffff>] 0xffffffff
    hardirq-on-W at:
                          [<c012f61f>] __lock_acquire+0x441/0xb93
                          [<c012fdd9>] lock_acquire+0x68/0x82
                          [<c02aaf2a>] _spin_lock_bh+0x30/0x3d
                          [<c02568b3>] dev_mc_upload+0x14/0x3a
                          [<c0253359>] dev_change_flags+0x31/0x10b
                          [<c028b237>] devinet_ioctl+0x235/0x546
                          [<c028b837>] inet_ioctl+0x89/0xaa
                          [<c0249cd1>] sock_ioctl+0x1ac/0x1ca
                          [<c0170a70>] do_ioctl+0x1c/0x53
                          [<c0170c93>] vfs_ioctl+0x1ec/0x203
                          [<c0170cf3>] sys_ioctl+0x49/0x62
                          [<c0103ca8>] sysenter_past_esp+0x5d/0x99
                          [<ffffffff>] 0xffffffff
  }
  ... key      at: [<c05f0788>] __key.29726+0x0/0x8
  -> (&rp->lock){++..} ops: 55 {
     initial-use  at:
                            [<c012f664>] __lock_acquire+0x486/0xb93
                            [<c012fdd9>] lock_acquire+0x68/0x82
                            [<c02ab22b>] _spin_lock_irqsave+0x34/0x44
                            [<d88a810a>] rhine_get_stats+0x2d/0x9d
[via_rhine]
                            [<c025ab78>] rtnl_fill_ifinfo+0x2ea/0x432
                            [<c025b0d7>] rtmsg_ifinfo+0x63/0xe0
                            [<c025b18f>] rtnetlink_event+0x3b/0x3f
                            [<c012020a>] notifier_call_chain+0x2b/0x4a
                            [<c0120294>] __raw_notifier_call_chain
+0x19/0x1f
                            [<c01202b4>] raw_notifier_call_chain
+0x1a/0x1c
                            [<c025304c>] register_netdevice+0x2e0/0x2ea
                            [<c0254284>] register_netdev+0x40/0x4d
                            [<d88a7c8c>] rhine_init_one+0x495/0x652
[via_rhine]
                            [<c01d5951>] pci_device_probe+0x3c/0x61
                            [<c0223684>] driver_probe_device+0xe8/0x168
                            [<c0223831>] __driver_attach+0x77/0xb1
                            [<c0222b5a>] bus_for_each_dev+0x3a/0x5f
                            [<c02234c7>] driver_attach+0x19/0x1b
                            [<c0222e7f>] bus_add_driver+0x76/0x17e
                            [<c0223a60>] driver_register+0x79/0x7f
                            [<c01d5ac6>] __pci_register_driver+0x5c/0x8c
                            [<d883c06c>] __ksymtab_ac97_bus_type
+0x0/0xfffffff6 [ac97_bus]
                            [<c01365eb>] sys_init_module+0x1252/0x138f
                            [<c0103ca8>] sysenter_past_esp+0x5d/0x99
                            [<ffffffff>] 0xffffffff
     in-hardirq-W at:
                            [<c012f58e>] __lock_acquire+0x3b0/0xb93
                            [<c012fdd9>] lock_acquire+0x68/0x82
                            [<c02aaeed>] _spin_lock+0x2b/0x38
                            [<d88a6b56>] rhine_interrupt+0x16a/0x696
[via_rhine]
                            [<c0142d2e>] handle_IRQ_event+0x1a/0x48
                            [<c014403e>] handle_level_irq+0x7c/0xc6
                            [<c01057eb>] do_IRQ+0x59/0x76
                            [<c0103f16>] common_interrupt+0x2e/0x34
                            [<c0123db9>] rcu_process_callbacks+0x15/0x26
                            [<c011a682>] tasklet_action+0x40/0x6c
                            [<c011a5a6>] __do_softirq+0x46/0x9c
                            [<c011a629>] do_softirq+0x2d/0x46
                            [<c011a9d5>] irq_exit+0x3b/0x6b
                            [<c01057f0>] do_IRQ+0x5e/0x76
                            [<c0103f16>] common_interrupt+0x2e/0x34
                            [<c02ab3b9>] error_code+0x71/0x78
                            [<ffffffff>] 0xffffffff
     in-softirq-W at:
                            [<c012f5b5>] __lock_acquire+0x3d7/0xb93
                            [<c012fdd9>] lock_acquire+0x68/0x82
                            [<c02ab158>] _spin_lock_irq+0x31/0x3e
                            [<d88a8614>] rhine_start_tx+0x1ad/0x244
[via_rhine]
                            [<c0253b4c>] dev_hard_start_xmit+0x216/0x274
                            [<c02606dc>] __qdisc_run+0xca/0x17f
                            [<c0255766>] dev_queue_xmit+0x155/0x26b
                            [<d8c52c05>] mld_sendpack+0x1f7/0x335 [ipv6]
                            [<d8c538ce>] mld_ifc_timer_expire
+0x1f6/0x21d [ipv6]
                            [<c011cbd6>] run_timer_softirq+0xf1/0x14e
                            [<c011a5a6>] __do_softirq+0x46/0x9c
                            [<c011a629>] do_softirq+0x2d/0x46
                            [<c011a9d5>] irq_exit+0x3b/0x6b
                            [<c01057f0>] do_IRQ+0x5e/0x76
                            [<c0103f16>] common_interrupt+0x2e/0x34
                            [<c02ab3b9>] error_code+0x71/0x78
                            [<ffffffff>] 0xffffffff
   }
   ... key      at: [<d88ac180>] __key.21685+0x0/0xffffc5fa [via_rhine]
  ... acquired at:
   [<c012fbc6>] __lock_acquire+0x9e8/0xb93
   [<c012fdd9>] lock_acquire+0x68/0x82
   [<c02ab158>] _spin_lock_irq+0x31/0x3e
   [<d88a8614>] rhine_start_tx+0x1ad/0x244 [via_rhine]
   [<c0253b4c>] dev_hard_start_xmit+0x216/0x274
   [<c02606dc>] __qdisc_run+0xca/0x17f
   [<c0255766>] dev_queue_xmit+0x155/0x26b
   [<d8c52c05>] mld_sendpack+0x1f7/0x335 [ipv6]
   [<d8c538ce>] mld_ifc_timer_expire+0x1f6/0x21d [ipv6]
   [<c011cbd6>] run_timer_softirq+0xf1/0x14e
   [<c011a5a6>] __do_softirq+0x46/0x9c
   [<c011a629>] do_softirq+0x2d/0x46
   [<c011a9d5>] irq_exit+0x3b/0x6b
   [<c01057f0>] do_IRQ+0x5e/0x76
   [<c0103f16>] common_interrupt+0x2e/0x34
   [<c02ab3b9>] error_code+0x71/0x78
   [<ffffffff>] 0xffffffff

 ... acquired at:
   [<c012fbc6>] __lock_acquire+0x9e8/0xb93
   [<c012fdd9>] lock_acquire+0x68/0x82
   [<c02aaf2a>] _spin_lock_bh+0x30/0x3d
   [<c02569c5>] dev_mc_add+0x33/0x141
   [<d8c539c0>] igmp6_group_added+0x58/0x120 [ipv6]
   [<d8c54175>] ipv6_dev_mc_inc+0x2f9/0x346 [ipv6]
   [<d8c42f7f>] ipv6_add_dev+0x232/0x240 [ipv6]
   [<d8c43503>] addrconf_notify+0x47/0x7d0 [ipv6]
   [<c0253ddc>] register_netdevice_notifier+0x35/0x5f
   [<d8c14336>] ____versions+0x1eb6/0xfffff9c8 [x_tables]
   [<d8c141d4>] ____versions+0x1d54/0xfffff9c8 [x_tables]
   [<c01365eb>] sys_init_module+0x1252/0x138f
   [<c0103ca8>] sysenter_past_esp+0x5d/0x99
   [<ffffffff>] 0xffffffff

 -> (&parent->list_lock){++..} ops: 14182 {
    initial-use  at:
                          [<c012f664>] __lock_acquire+0x486/0xb93
                          [<c012fdd9>] lock_acquire+0x68/0x82
                          [<c02aaeed>] _spin_lock+0x2b/0x38
                          [<c01637a4>] cache_alloc_refill+0x4f/0x44f
                          [<c0163d52>] kmem_cache_zalloc+0x50/0x97
                          [<c01643cc>] kmem_cache_create+0x166/0x447
                          [<c039804a>] kmem_cache_init+0x160/0x3c7
                          [<c0388ab4>] start_kernel+0x217/0x38d
                          [<00000000>] 0x0
                          [<ffffffff>] 0xffffffff
    in-hardirq-W at:
                          [<c012f58e>] __lock_acquire+0x3b0/0xb93
                          [<c012fdd9>] lock_acquire+0x68/0x82
                          [<c02aaeed>] _spin_lock+0x2b/0x38
                          [<c01630e3>] cache_flusharray+0x22/0xa5
                          [<c0163254>] kmem_cache_free+0x62/0x8e
                          [<c014c6cb>] mempool_free_slab+0x11/0x13
                          [<c014c731>] mempool_free+0x64/0x6c
                          [<c0185212>] bio_free+0x2d/0x31
                          [<c0185227>] bio_fs_destructor+0x11/0x13
                          [<c018534c>] bio_put+0x28/0x2a
                          [<c01839f2>] end_bio_bh_io_sync+0x39/0x3f
                          [<c018527c>] bio_endio+0x53/0x5b
                          [<c01b49fa>] __end_that_request_first
+0xcf/0x35d
                          [<c01b4ca3>] end_that_request_first+0xe/0x10
                          [<d895ec75>] ide_end_request+0x87/0xd1
[ide_core]
                          [<d8965d80>] ide_dma_intr+0x58/0x98 [ide_core]
                          [<d895f7fe>] ide_intr+0x16b/0x1cf [ide_core]
                          [<c0142d2e>] handle_IRQ_event+0x1a/0x48
                          [<c014403e>] handle_level_irq+0x7c/0xc6
                          [<c01057eb>] do_IRQ+0x59/0x76
                          [<c0103f16>] common_interrupt+0x2e/0x34
                          [<c010239b>] cpu_idle+0x44/0x5e
                          [<c0101110>] rest_init+0x21/0x25
                          [<c0388c22>] start_kernel+0x385/0x38d
                          [<00000000>] 0x0
                          [<ffffffff>] 0xffffffff
    in-softirq-W at:
                          [<c012f5b5>] __lock_acquire+0x3d7/0xb93
                          [<c012fdd9>] lock_acquire+0x68/0x82
                          [<c02aaeed>] _spin_lock+0x2b/0x38
                          [<c01630e3>] cache_flusharray+0x22/0xa5
                          [<c01631c4>] kfree+0x5e/0x8c
                          [<c0114875>] free_task+0x11/0x21
                          [<c01160c3>] __put_task_struct+0x10f/0x118
                          [<c01178cd>] delayed_put_task_struct+0x3e/0x45
                          [<c0123d3b>] __rcu_process_callbacks
+0x103/0x16c
                          [<c0123db9>] rcu_process_callbacks+0x15/0x26
                          [<c011a682>] tasklet_action+0x40/0x6c
                          [<c011a5a6>] __do_softirq+0x46/0x9c
                          [<c011a629>] do_softirq+0x2d/0x46
                          [<c011a9d5>] irq_exit+0x3b/0x6b
                          [<c01057f0>] do_IRQ+0x5e/0x76
                          [<c0103f16>] common_interrupt+0x2e/0x34
                          [<c01ed01e>] acpi_ps_alloc_op+0x5b/0x98
                          [<c01ecc85>] acpi_ps_parse_loop+0x6d9/0x714
                          [<c01ebf86>] acpi_ps_parse_aml+0x68/0x246
                          [<c01ed236>] acpi_ps_execute_method
+0x11f/0x1c1
                          [<c01ea44c>] acpi_ns_evaluate+0xa0/0x100
                          [<c01efba8>] acpi_ut_evaluate_object
+0x52/0x18c
                          [<c01e226e>] acpi_ev_match_prw_and_gpe
+0x1f/0xbe
                          [<c01eb6b9>] acpi_ns_walk_namespace+0xf9/0x114
                          [<c01e2216>] acpi_ev_initialize_gpe_block
+0x4d/0x86
                          [<c01dfe20>] acpi_ev_install_fadt_gpes
+0x27/0x47
                          [<c01f0ace>] acpi_enable_subsystem+0x83/0x8f
                          [<c039af8c>] acpi_init+0x57/0x1cd
                          [<c038872c>] kernel_init+0xb9/0x22a
                          [<c0104007>] kernel_thread_helper+0x7/0x10
                          [<ffffffff>] 0xffffffff
  }
  ... key      at: [<c05e5a88>] __key.17030+0x0/0x8
  -> (&zone->lock){++..} ops: 6654 {
     initial-use  at:
                            [<c012f664>] __lock_acquire+0x486/0xb93
                            [<c012fdd9>] lock_acquire+0x68/0x82
                            [<c02aaeed>] _spin_lock+0x2b/0x38
                            [<c014dbc8>] free_pages_bulk+0x1a/0x195
                            [<c014de55>] free_hot_cold_page+0x112/0x156
                            [<c014dee4>] free_hot_page+0xd/0xf
                            [<c014df1e>] __free_pages+0x38/0x4a
                            [<c03977a4>] __free_pages_bootmem+0x61/0x65
                            [<c0395fb4>] free_all_bootmem_core
+0xd6/0x147
                            [<c0396035>] free_all_bootmem+0x10/0x12
                            [<c0393631>] mem_init+0x2a/0x2d0
                            [<c0388aaf>] start_kernel+0x212/0x38d
                            [<00000000>] 0x0
                            [<ffffffff>] 0xffffffff
     in-hardirq-W at:
                            [<c012f58e>] __lock_acquire+0x3b0/0xb93
                            [<c012fdd9>] lock_acquire+0x68/0x82
                            [<c02aaeed>] _spin_lock+0x2b/0x38
                            [<c014dbc8>] free_pages_bulk+0x1a/0x195
                            [<c014de55>] free_hot_cold_page+0x112/0x156
                            [<c014dee4>] free_hot_page+0xd/0xf
                            [<c014df1e>] __free_pages+0x38/0x4a
                            [<c014df50>] free_pages+0x20/0x22
                            [<c0163060>] kmem_freepages+0x9c/0xa4
                            [<c016331e>] slab_destroy+0x61/0x79
                            [<c01633ed>] free_block+0xb7/0xf4
                            [<c0163138>] cache_flusharray+0x77/0xa5
                            [<c0163254>] kmem_cache_free+0x62/0x8e
                            [<c014c6cb>] mempool_free_slab+0x11/0x13
                            [<c014c731>] mempool_free+0x64/0x6c
                            [<c0185212>] bio_free+0x2d/0x31
                            [<c0185227>] bio_fs_destructor+0x11/0x13
                            [<c018534c>] bio_put+0x28/0x2a
                            [<c01839f2>] end_bio_bh_io_sync+0x39/0x3f
                            [<c018527c>] bio_endio+0x53/0x5b
                            [<c01b49fa>] __end_that_request_first
+0xcf/0x35d
                            [<c01b4ca3>] end_that_request_first+0xe/0x10
                            [<d895ec75>] ide_end_request+0x87/0xd1
[ide_core]
                            [<d8965d80>] ide_dma_intr+0x58/0x98
[ide_core]
                            [<d895f7fe>] ide_intr+0x16b/0x1cf [ide_core]
                            [<c0142d2e>] handle_IRQ_event+0x1a/0x48
                            [<c014403e>] handle_level_irq+0x7c/0xc6
                            [<c01057eb>] do_IRQ+0x59/0x76
                            [<c0103f16>] common_interrupt+0x2e/0x34
                            [<c010239b>] cpu_idle+0x44/0x5e
                            [<c0101110>] rest_init+0x21/0x25
                            [<c0388c22>] start_kernel+0x385/0x38d
                            [<00000000>] 0x0
                            [<ffffffff>] 0xffffffff
     in-softirq-W at:
                            [<c012f5b5>] __lock_acquire+0x3d7/0xb93
                            [<c012fdd9>] lock_acquire+0x68/0x82
                            [<c02aaeed>] _spin_lock+0x2b/0x38
                            [<c014da1c>] __free_pages_ok+0xf6/0x278
                            [<c014df29>] __free_pages+0x43/0x4a
                            [<c014df50>] free_pages+0x20/0x22
                            [<c0163060>] kmem_freepages+0x9c/0xa4
                            [<c016331e>] slab_destroy+0x61/0x79
                            [<c01633ed>] free_block+0xb7/0xf4
                            [<c0163138>] cache_flusharray+0x77/0xa5
                            [<c01631c4>] kfree+0x5e/0x8c
                            [<c0114875>] free_task+0x11/0x21
                            [<c01160c3>] __put_task_struct+0x10f/0x118
                            [<c01178cd>] delayed_put_task_struct
+0x3e/0x45
                            [<c0123d3b>] __rcu_process_callbacks
+0x103/0x16c
                            [<c0123db9>] rcu_process_callbacks+0x15/0x26
                            [<c011a682>] tasklet_action+0x40/0x6c
                            [<c011a5a6>] __do_softirq+0x46/0x9c
                            [<c011a629>] do_softirq+0x2d/0x46
                            [<c011a9d5>] irq_exit+0x3b/0x6b
                            [<c01057f0>] do_IRQ+0x5e/0x76
                            [<c0103f16>] common_interrupt+0x2e/0x34
                            [<c02a9e79>] __mutex_lock_slowpath
+0x238/0x240
                            [<c02a9ea5>] mutex_lock+0x24/0x28
                            [<c016c7b2>] do_lookup+0x69/0x13d
                            [<c016e6d6>] __link_path_walk+0x814/0xc69
                            [<c016eb74>] link_path_walk+0x49/0xbc
                            [<c016ee96>] do_path_lookup+0x18a/0x1dc
                            [<c016f822>] __path_lookup_intent_open
+0x44/0x75
                            [<c016f8c6>] path_lookup_open+0x21/0x27
                            [<c0169bc9>] open_exec+0x27/0xa5
                            [<c016aeaa>] do_execve+0x36/0x1f1
                            [<c01021a3>] sys_execve+0x2e/0x50
                            [<c0103d34>] syscall_call+0x7/0xb
                            [<ffffffff>] 0xffffffff
   }
   ... key      at: [<c05de598>] __key.20983+0x0/0x8
  ... acquired at:
   [<c012fbc6>] __lock_acquire+0x9e8/0xb93
   [<c012fdd9>] lock_acquire+0x68/0x82
   [<c02aaeed>] _spin_lock+0x2b/0x38
   [<c014da1c>] __free_pages_ok+0xf6/0x278
   [<c014df29>] __free_pages+0x43/0x4a
   [<c014df50>] free_pages+0x20/0x22
   [<c0163060>] kmem_freepages+0x9c/0xa4
   [<c016331e>] slab_destroy+0x61/0x79
   [<c01633ed>] free_block+0xb7/0xf4
   [<c0163138>] cache_flusharray+0x77/0xa5
   [<c01631c4>] kfree+0x5e/0x8c
   [<c0114875>] free_task+0x11/0x21
   [<c01160c3>] __put_task_struct+0x10f/0x118
   [<c01178cd>] delayed_put_task_struct+0x3e/0x45
   [<c0123d3b>] __rcu_process_callbacks+0x103/0x16c
   [<c0123db9>] rcu_process_callbacks+0x15/0x26
   [<c011a682>] tasklet_action+0x40/0x6c
   [<c011a5a6>] __do_softirq+0x46/0x9c
   [<c011a629>] do_softirq+0x2d/0x46
   [<c011a9d5>] irq_exit+0x3b/0x6b
   [<c01057f0>] do_IRQ+0x5e/0x76
   [<c0103f16>] common_interrupt+0x2e/0x34
   [<c02a9e79>] __mutex_lock_slowpath+0x238/0x240
   [<c02a9ea5>] mutex_lock+0x24/0x28
   [<c016c7b2>] do_lookup+0x69/0x13d
   [<c016e6d6>] __link_path_walk+0x814/0xc69
   [<c016eb74>] link_path_walk+0x49/0xbc
   [<c016ee96>] do_path_lookup+0x18a/0x1dc
   [<c016f822>] __path_lookup_intent_open+0x44/0x75
   [<c016f8c6>] path_lookup_open+0x21/0x27
   [<c0169bc9>] open_exec+0x27/0xa5
   [<c016aeaa>] do_execve+0x36/0x1f1
   [<c01021a3>] sys_execve+0x2e/0x50
   [<c0103d34>] syscall_call+0x7/0xb
   [<ffffffff>] 0xffffffff

  -> (pgd_lock){....} ops: 119 {
     initial-use  at:
                            [<c012f664>] __lock_acquire+0x486/0xb93
                            [<c012fdd9>] lock_acquire+0x68/0x82
                            [<c02ab22b>] _spin_lock_irqsave+0x34/0x44
                            [<c0110d8a>] pgd_ctor+0x25/0x7d
                            [<c0163aee>] cache_alloc_refill+0x399/0x44f
                            [<c016372d>] kmem_cache_alloc+0x50/0x78
                            [<c0110d1f>] pgd_alloc+0x15/0x17
                            [<c0114691>] mm_init+0xa6/0xcd
                            [<c0115bbd>] mm_alloc+0x2e/0x32
                            [<c016aedd>] do_execve+0x69/0x1f1
                            [<c01021a3>] sys_execve+0x2e/0x50
                            [<c0103d34>] syscall_call+0x7/0xb
                            [<ffffffff>] 0xffffffff
   }
   ... key      at: [<c0361cfc>] pgd_lock+0x1c/0x40
  ... acquired at:
   [<c012fbc6>] __lock_acquire+0x9e8/0xb93
   [<c012fdd9>] lock_acquire+0x68/0x82
   [<c02ab22b>] _spin_lock_irqsave+0x34/0x44
   [<c0110d34>] pgd_dtor+0x13/0x44
   [<c01632f0>] slab_destroy+0x33/0x79
   [<c01633ed>] free_block+0xb7/0xf4
   [<c0163138>] cache_flusharray+0x77/0xa5
   [<c0163254>] kmem_cache_free+0x62/0x8e
   [<c0110d08>] pgd_free+0x12/0x14
   [<c011477d>] __mmdrop+0x1c/0x33
   [<c02a8dce>] __sched_text_start+0x546/0x59f
   [<c01188da>] do_wait+0x90e/0xa04
   [<c0118a00>] sys_wait4+0x30/0x32
   [<c0118a29>] sys_waitpid+0x27/0x29
   [<c0103ca8>] sysenter_past_esp+0x5d/0x99
   [<ffffffff>] 0xffffffff

 ... acquired at:
   [<c012fbc6>] __lock_acquire+0x9e8/0xb93
   [<c012fdd9>] lock_acquire+0x68/0x82
   [<c02aaeed>] _spin_lock+0x2b/0x38
   [<c01637a4>] cache_alloc_refill+0x4f/0x44f
   [<c0163c19>] __kmalloc+0x75/0x9a
   [<c024f856>] __alloc_skb+0x6b/0x11b
   [<c024c162>] sock_alloc_send_skb+0x7c/0x1a9
   [<d8c52299>] mld_newpack+0x5d/0x172 [ipv6]
   [<d8c523d2>] add_grhead+0x24/0xa4 [ipv6]
   [<d8c53072>] add_grec+0x32f/0x35a [ipv6]
   [<d8c5389f>] mld_ifc_timer_expire+0x1c7/0x21d [ipv6]
   [<c011cbd6>] run_timer_softirq+0xf1/0x14e
   [<c011a5a6>] __do_softirq+0x46/0x9c
   [<c011a629>] do_softirq+0x2d/0x46
   [<c011a9d5>] irq_exit+0x3b/0x6b
   [<c01057f0>] do_IRQ+0x5e/0x76
   [<c0103f16>] common_interrupt+0x2e/0x34
   [<c02ab3b9>] error_code+0x71/0x78
   [<ffffffff>] 0xffffffff


the soft-irq-unsafe lock's dependencies:
-> (&priv->lock){--..} ops: 2 {
   initial-use  at:
                        [<c012f664>] __lock_acquire+0x486/0xb93
                        [<c012fdd9>] lock_acquire+0x68/0x82
                        [<c02aaeed>] _spin_lock+0x2b/0x38
                        [<d886b729>] sc92031_open+0xcc/0x16f [sc92031]
                        [<c02549e7>] dev_open+0x33/0x6e
                        [<c025337f>] dev_change_flags+0x57/0x10b
                        [<c028b237>] devinet_ioctl+0x235/0x546
                        [<c028b837>] inet_ioctl+0x89/0xaa
                        [<c0249cd1>] sock_ioctl+0x1ac/0x1ca
                        [<c0170a70>] do_ioctl+0x1c/0x53
                        [<c0170c93>] vfs_ioctl+0x1ec/0x203
                        [<c0170cf3>] sys_ioctl+0x49/0x62
                        [<c0103ca8>] sysenter_past_esp+0x5d/0x99
                        [<ffffffff>] 0xffffffff
   softirq-on-W at:
                        [<c012f649>] __lock_acquire+0x46b/0xb93
                        [<c012fdd9>] lock_acquire+0x68/0x82
                        [<c02aaeed>] _spin_lock+0x2b/0x38
                        [<d886b729>] sc92031_open+0xcc/0x16f [sc92031]
                        [<c02549e7>] dev_open+0x33/0x6e
                        [<c025337f>] dev_change_flags+0x57/0x10b
                        [<c028b237>] devinet_ioctl+0x235/0x546
                        [<c028b837>] inet_ioctl+0x89/0xaa
                        [<c0249cd1>] sock_ioctl+0x1ac/0x1ca
                        [<c0170a70>] do_ioctl+0x1c/0x53
                        [<c0170c93>] vfs_ioctl+0x1ec/0x203
                        [<c0170cf3>] sys_ioctl+0x49/0x62
                        [<c0103ca8>] sysenter_past_esp+0x5d/0x99
                        [<ffffffff>] 0xffffffff
   hardirq-on-W at:
                        [<c012f61f>] __lock_acquire+0x441/0xb93
                        [<c012fdd9>] lock_acquire+0x68/0x82
                        [<c02aaeed>] _spin_lock+0x2b/0x38
                        [<d886b729>] sc92031_open+0xcc/0x16f [sc92031]
                        [<c02549e7>] dev_open+0x33/0x6e
                        [<c025337f>] dev_change_flags+0x57/0x10b
                        [<c028b237>] devinet_ioctl+0x235/0x546
                        [<c028b837>] inet_ioctl+0x89/0xaa
                        [<c0249cd1>] sock_ioctl+0x1ac/0x1ca
                        [<c0170a70>] do_ioctl+0x1c/0x53
                        [<c0170c93>] vfs_ioctl+0x1ec/0x203
                        [<c0170cf3>] sys_ioctl+0x49/0x62
                        [<c0103ca8>] sysenter_past_esp+0x5d/0x99
                        [<ffffffff>] 0xffffffff
 }
 ... key      at: [<d886e180>] __key.22056+0x0/0xffffd670 [sc92031]
 -> (lweventlist_lock){....} ops: 1 {
    initial-use  at:
                          [<c012f664>] __lock_acquire+0x486/0xb93
                          [<c012fdd9>] lock_acquire+0x68/0x82
                          [<c02ab22b>] _spin_lock_irqsave+0x34/0x44
                          [<c025c17a>] linkwatch_fire_event+0x58/0xae
                          [<c02605e3>] netif_carrier_off+0x1a/0x1c
                          [<d886a650>] _sc92031_reset+0x1a5/0x1dd
[sc92031]
                          [<d886b730>] sc92031_open+0xd3/0x16f [sc92031]
                          [<c02549e7>] dev_open+0x33/0x6e
                          [<c025337f>] dev_change_flags+0x57/0x10b
                          [<c028b237>] devinet_ioctl+0x235/0x546
                          [<c028b837>] inet_ioctl+0x89/0xaa
                          [<c0249cd1>] sock_ioctl+0x1ac/0x1ca
                          [<c0170a70>] do_ioctl+0x1c/0x53
                          [<c0170c93>] vfs_ioctl+0x1ec/0x203
                          [<c0170cf3>] sys_ioctl+0x49/0x62
                          [<c0103ca8>] sysenter_past_esp+0x5d/0x99
                          [<ffffffff>] 0xffffffff
  }
  ... key      at: [<c037a19c>] lweventlist_lock+0x1c/0x28
 ... acquired at:
   [<c012fbc6>] __lock_acquire+0x9e8/0xb93
   [<c012fdd9>] lock_acquire+0x68/0x82
   [<c02ab22b>] _spin_lock_irqsave+0x34/0x44
   [<c025c17a>] linkwatch_fire_event+0x58/0xae
   [<c02605e3>] netif_carrier_off+0x1a/0x1c
   [<d886a650>] _sc92031_reset+0x1a5/0x1dd [sc92031]
   [<d886b730>] sc92031_open+0xd3/0x16f [sc92031]
   [<c02549e7>] dev_open+0x33/0x6e
   [<c025337f>] dev_change_flags+0x57/0x10b
   [<c028b237>] devinet_ioctl+0x235/0x546
   [<c028b837>] inet_ioctl+0x89/0xaa
   [<c0249cd1>] sock_ioctl+0x1ac/0x1ca
   [<c0170a70>] do_ioctl+0x1c/0x53
   [<c0170c93>] vfs_ioctl+0x1ec/0x203
   [<c0170cf3>] sys_ioctl+0x49/0x62
   [<c0103ca8>] sysenter_past_esp+0x5d/0x99
   [<ffffffff>] 0xffffffff

 -> (&cwq->lock){++..} ops: 8986 {
    initial-use  at:
                          [<c012f664>] __lock_acquire+0x486/0xb93
                          [<c012fdd9>] lock_acquire+0x68/0x82
                          [<c02ab22b>] _spin_lock_irqsave+0x34/0x44
                          [<c0123109>] __queue_work+0x12/0x33
                          [<c0123152>] queue_work+0x28/0x33
                          [<c012293e>] call_usermodehelper_keys
+0xc5/0xea
                          [<c01bf6ca>] kobject_uevent_env+0x3d3/0x42b
                          [<c01bf72f>] kobject_uevent+0xd/0x12
                          [<c022184a>] device_add+0x32f/0x518
                          [<c0221a48>] device_register+0x15/0x19
                          [<c0221faa>] device_create+0x77/0x99
                          [<c039e0d0>] vtconsole_class_init+0x71/0xb5
                          [<c038872c>] kernel_init+0xb9/0x22a
                          [<c0104007>] kernel_thread_helper+0x7/0x10
                          [<ffffffff>] 0xffffffff
    in-hardirq-W at:
                          [<c012f58e>] __lock_acquire+0x3b0/0xb93
                          [<c012fdd9>] lock_acquire+0x68/0x82
                          [<c02ab22b>] _spin_lock_irqsave+0x34/0x44
                          [<c0123109>] __queue_work+0x12/0x33
                          [<c0123152>] queue_work+0x28/0x33
                          [<c012316f>] schedule_work+0x12/0x14
                          [<c0229e72>] schedule_bh+0x15/0x17
                          [<c0229ff8>] floppy_interrupt+0x168/0x192
                          [<c022d014>] floppy_hardint+0x1c/0xbc
                          [<c0142d2e>] handle_IRQ_event+0x1a/0x48
                          [<c014403e>] handle_level_irq+0x7c/0xc6
                          [<c01057eb>] do_IRQ+0x59/0x76
                          [<c0103f16>] common_interrupt+0x2e/0x34
                          [<c010239b>] cpu_idle+0x44/0x5e
                          [<c0101110>] rest_init+0x21/0x25
                          [<c0388c22>] start_kernel+0x385/0x38d
                          [<00000000>] 0x0
                          [<ffffffff>] 0xffffffff
    in-softirq-W at:
                          [<c012f5b5>] __lock_acquire+0x3d7/0xb93
                          [<c012fdd9>] lock_acquire+0x68/0x82
                          [<c02ab22b>] _spin_lock_irqsave+0x34/0x44
                          [<c0123109>] __queue_work+0x12/0x33
                          [<c01234b5>] delayed_work_timer_fn+0x17/0x19
                          [<c011cbd6>] run_timer_softirq+0xf1/0x14e
                          [<c011a5a6>] __do_softirq+0x46/0x9c
                          [<c011a629>] do_softirq+0x2d/0x46
                          [<c011a9d5>] irq_exit+0x3b/0x6b
                          [<c01057f0>] do_IRQ+0x5e/0x76
                          [<c0103f16>] common_interrupt+0x2e/0x34
                          [<c010239b>] cpu_idle+0x44/0x5e
                          [<c0101110>] rest_init+0x21/0x25
                          [<c0388c22>] start_kernel+0x385/0x38d
                          [<00000000>] 0x0
                          [<ffffffff>] 0xffffffff
  }
  ... key      at: [<c03e22e0>] __key.10619+0x0/0x8
  -> (&q->lock){++..} ops: 54469 {
     initial-use  at:
                            [<c012f664>] __lock_acquire+0x486/0xb93
                            [<c012fdd9>] lock_acquire+0x68/0x82
                            [<c02ab158>] _spin_lock_irq+0x31/0x3e
                            [<c02a8fb9>] wait_for_completion+0x29/0xaf
                            [<c012599d>] keventd_create_kthread
+0x32/0x62
                            [<c0125a30>] kthread_create+0x63/0x9e
                            [<c0394848>] cpu_callback+0x35/0x7a
                            [<c03948a4>] spawn_ksoftirqd+0x17/0x37
                            [<c03886ac>] kernel_init+0x39/0x22a
                            [<c0104007>] kernel_thread_helper+0x7/0x10
                            [<ffffffff>] 0xffffffff
     in-hardirq-W at:
                            [<c012f58e>] __lock_acquire+0x3b0/0xb93
                            [<c012fdd9>] lock_acquire+0x68/0x82
                            [<c02ab22b>] _spin_lock_irqsave+0x34/0x44
                            [<c0112ae3>] __wake_up+0x15/0x42
                            [<c0122d39>] insert_work+0x64/0x6b
                            [<c0123119>] __queue_work+0x22/0x33
                            [<c0123152>] queue_work+0x28/0x33
                            [<c012316f>] schedule_work+0x12/0x14
                            [<c0229e72>] schedule_bh+0x15/0x17
                            [<c0229ff8>] floppy_interrupt+0x168/0x192
                            [<c022d014>] floppy_hardint+0x1c/0xbc
                            [<c0142d2e>] handle_IRQ_event+0x1a/0x48
                            [<c014403e>] handle_level_irq+0x7c/0xc6
                            [<c01057eb>] do_IRQ+0x59/0x76
                            [<c0103f16>] common_interrupt+0x2e/0x34
                            [<c010239b>] cpu_idle+0x44/0x5e
                            [<c0101110>] rest_init+0x21/0x25
                            [<c0388c22>] start_kernel+0x385/0x38d
                            [<00000000>] 0x0
                            [<ffffffff>] 0xffffffff
     in-softirq-W at:
                            [<c012f5b5>] __lock_acquire+0x3d7/0xb93
                            [<c012fdd9>] lock_acquire+0x68/0x82
                            [<c02ab22b>] _spin_lock_irqsave+0x34/0x44
                            [<c0112a4f>] complete+0x15/0x4a
                            [<c0123b65>] wakeme_after_rcu+0xe/0x10
                            [<c0123d3b>] __rcu_process_callbacks
+0x103/0x16c
                            [<c0123db9>] rcu_process_callbacks+0x15/0x26
                            [<c011a682>] tasklet_action+0x40/0x6c
                            [<c011a5a6>] __do_softirq+0x46/0x9c
                            [<c011a629>] do_softirq+0x2d/0x46
                            [<c011a9d5>] irq_exit+0x3b/0x6b
                            [<c01057f0>] do_IRQ+0x5e/0x76
                            [<c0103f16>] common_interrupt+0x2e/0x34
                            [<c010239b>] cpu_idle+0x44/0x5e
                            [<c0101110>] rest_init+0x21/0x25
                            [<c0388c22>] start_kernel+0x385/0x38d
                            [<00000000>] 0x0
                            [<ffffffff>] 0xffffffff
   }
   ... key      at: [<c03e271c>] __key.14042+0x0/0x8
   -> (&rq->rq_lock_key){++..} ops: 67981 {
      initial-use  at:
                              [<c012f664>] __lock_acquire+0x486/0xb93
                              [<c012fdd9>] lock_acquire+0x68/0x82
                              [<c02ab22b>] _spin_lock_irqsave+0x34/0x44
                              [<c0393e8d>] init_idle+0x5c/0x86
                              [<c03940cb>] sched_init+0x214/0x21c
                              [<c038897a>] start_kernel+0xdd/0x38d
                              [<00000000>] 0x0
                              [<ffffffff>] 0xffffffff
      in-hardirq-W at:
                              [<c012f58e>] __lock_acquire+0x3b0/0xb93
                              [<c012fdd9>] lock_acquire+0x68/0x82
                              [<c02aaeed>] _spin_lock+0x2b/0x38
                              [<c0114195>] scheduler_tick+0x8e/0xbd
                              [<c011d1bf>] update_process_times
+0x4f/0x5a
                              [<c012bf0d>] tick_periodic+0x69/0x75
                              [<c012bf2b>] tick_handle_periodic
+0x12/0x63
                              [<c01066d9>] timer_interrupt+0x11/0x18
                              [<c0142d2e>] handle_IRQ_event+0x1a/0x48
                              [<c014403e>] handle_level_irq+0x7c/0xc6
                              [<c01057eb>] do_IRQ+0x59/0x76
                              [<c0103f16>] common_interrupt+0x2e/0x34
                              [<c038f206>] request_standard_resources
+0x75/0x2fc
                              [<c038872c>] kernel_init+0xb9/0x22a
                              [<c0104007>] kernel_thread_helper+0x7/0x10
                              [<ffffffff>] 0xffffffff
      in-softirq-W at:
                              [<c012f5b5>] __lock_acquire+0x3d7/0xb93
                              [<c012fdd9>] lock_acquire+0x68/0x82
                              [<c02aaeed>] _spin_lock+0x2b/0x38
                              [<c0113260>] task_rq_lock+0x1a/0x21
                              [<c0113944>] try_to_wake_up+0x18/0xa0
                              [<c01139da>] default_wake_function
+0xe/0x10
                              [<c01127f0>] __wake_up_common+0x32/0x55
                              [<c0112a73>] complete+0x39/0x4a
                              [<c0123b65>] wakeme_after_rcu+0xe/0x10
                              [<c0123d3b>] __rcu_process_callbacks
+0x103/0x16c
                              [<c0123db9>] rcu_process_callbacks
+0x15/0x26
                              [<c011a682>] tasklet_action+0x40/0x6c
                              [<c011a5a6>] __do_softirq+0x46/0x9c
                              [<c011a629>] do_softirq+0x2d/0x46
                              [<c011a9d5>] irq_exit+0x3b/0x6b
                              [<c01057f0>] do_IRQ+0x5e/0x76
                              [<c0103f16>] common_interrupt+0x2e/0x34
                              [<c010239b>] cpu_idle+0x44/0x5e
                              [<c0101110>] rest_init+0x21/0x25
                              [<c0388c22>] start_kernel+0x385/0x38d
                              [<00000000>] 0x0
                              [<ffffffff>] 0xffffffff
    }
    ... key      at: [<c03bfd38>] per_cpu__runqueues+0x9f8/0xa00
   ... acquired at:
   [<c012fbc6>] __lock_acquire+0x9e8/0xb93
   [<c012fdd9>] lock_acquire+0x68/0x82
   [<c02aaeed>] _spin_lock+0x2b/0x38
   [<c0113260>] task_rq_lock+0x1a/0x21
   [<c0113944>] try_to_wake_up+0x18/0xa0
   [<c01139da>] default_wake_function+0xe/0x10
   [<c01127f0>] __wake_up_common+0x32/0x55
   [<c0112a73>] complete+0x39/0x4a
   [<c0125af2>] kthread+0x87/0xcd
   [<c0104007>] kernel_thread_helper+0x7/0x10
   [<ffffffff>] 0xffffffff

  ... acquired at:
   [<c012fbc6>] __lock_acquire+0x9e8/0xb93
   [<c012fdd9>] lock_acquire+0x68/0x82
   [<c02ab22b>] _spin_lock_irqsave+0x34/0x44
   [<c0112ae3>] __wake_up+0x15/0x42
   [<c0122d39>] insert_work+0x64/0x6b
   [<c0123119>] __queue_work+0x22/0x33
   [<c0123152>] queue_work+0x28/0x33
   [<c012293e>] call_usermodehelper_keys+0xc5/0xea
   [<c01bf6ca>] kobject_uevent_env+0x3d3/0x42b
   [<c01bf72f>] kobject_uevent+0xd/0x12
   [<c022184a>] device_add+0x32f/0x518
   [<c0221a48>] device_register+0x15/0x19
   [<c0221faa>] device_create+0x77/0x99
   [<c039e0d0>] vtconsole_class_init+0x71/0xb5
   [<c038872c>] kernel_init+0xb9/0x22a
   [<c0104007>] kernel_thread_helper+0x7/0x10
   [<ffffffff>] 0xffffffff

 ... acquired at:
   [<c012fbc6>] __lock_acquire+0x9e8/0xb93
   [<c012fdd9>] lock_acquire+0x68/0x82
   [<c02ab22b>] _spin_lock_irqsave+0x34/0x44
   [<c0123109>] __queue_work+0x12/0x33
   [<c0123152>] queue_work+0x28/0x33
   [<c0123315>] queue_delayed_work+0xf/0x22
   [<c012333c>] schedule_delayed_work+0x14/0x16
   [<c025c1cb>] linkwatch_fire_event+0xa9/0xae
   [<c02605e3>] netif_carrier_off+0x1a/0x1c
   [<d886a650>] _sc92031_reset+0x1a5/0x1dd [sc92031]
   [<d886b730>] sc92031_open+0xd3/0x16f [sc92031]
   [<c02549e7>] dev_open+0x33/0x6e
   [<c025337f>] dev_change_flags+0x57/0x10b
   [<c028b237>] devinet_ioctl+0x235/0x546
   [<c028b837>] inet_ioctl+0x89/0xaa
   [<c0249cd1>] sock_ioctl+0x1ac/0x1ca
   [<c0170a70>] do_ioctl+0x1c/0x53
   [<c0170c93>] vfs_ioctl+0x1ec/0x203
   [<c0170cf3>] sys_ioctl+0x49/0x62
   [<c0103ca8>] sysenter_past_esp+0x5d/0x99
   [<ffffffff>] 0xffffffff

 -> (ratelimit_lock){....} ops: 1 {
    initial-use  at:
                          [<c012f664>] __lock_acquire+0x486/0xb93
                          [<c012fdd9>] lock_acquire+0x68/0x82
                          [<c02ab22b>] _spin_lock_irqsave+0x34/0x44
                          [<c0116ec8>] __printk_ratelimit+0x1d/0xa5
                          [<c0116f66>] printk_ratelimit+0x16/0x18
                          [<d886a489>] _sc92031_check_media+0x160/0x182
[sc92031]
                          [<d886a660>] _sc92031_reset+0x1b5/0x1dd
[sc92031]
                          [<d886b730>] sc92031_open+0xd3/0x16f [sc92031]
                          [<c02549e7>] dev_open+0x33/0x6e
                          [<c025337f>] dev_change_flags+0x57/0x10b
                          [<c028b237>] devinet_ioctl+0x235/0x546
                          [<c028b837>] inet_ioctl+0x89/0xaa
                          [<c0249cd1>] sock_ioctl+0x1ac/0x1ca
                          [<c0170a70>] do_ioctl+0x1c/0x53
                          [<c0170c93>] vfs_ioctl+0x1ec/0x203
                          [<c0170cf3>] sys_ioctl+0x49/0x62
                          [<c0103ca8>] sysenter_past_esp+0x5d/0x99
                          [<ffffffff>] 0xffffffff
  }
  ... key      at: [<c03620dc>] ratelimit_lock.21472+0x1c/0x28
 ... acquired at:
   [<c012fbc6>] __lock_acquire+0x9e8/0xb93
   [<c012fdd9>] lock_acquire+0x68/0x82
   [<c02ab22b>] _spin_lock_irqsave+0x34/0x44
   [<c0116ec8>] __printk_ratelimit+0x1d/0xa5
   [<c0116f66>] printk_ratelimit+0x16/0x18
   [<d886a489>] _sc92031_check_media+0x160/0x182 [sc92031]
   [<d886a660>] _sc92031_reset+0x1b5/0x1dd [sc92031]
   [<d886b730>] sc92031_open+0xd3/0x16f [sc92031]
   [<c02549e7>] dev_open+0x33/0x6e
   [<c025337f>] dev_change_flags+0x57/0x10b
   [<c028b237>] devinet_ioctl+0x235/0x546
   [<c028b837>] inet_ioctl+0x89/0xaa
   [<c0249cd1>] sock_ioctl+0x1ac/0x1ca
   [<c0170a70>] do_ioctl+0x1c/0x53
   [<c0170c93>] vfs_ioctl+0x1ec/0x203
   [<c0170cf3>] sys_ioctl+0x49/0x62
   [<c0103ca8>] sysenter_past_esp+0x5d/0x99
   [<ffffffff>] 0xffffffff


stack backtrace:
 [<c010433c>] show_trace_log_lvl+0x1a/0x30
 [<c0104e92>] show_trace+0x12/0x14
 [<c0104eeb>] dump_stack+0x16/0x18
 [<c012f13b>] check_usage+0x25a/0x264
 [<c012fb28>] __lock_acquire+0x94a/0xb93
 [<c012fdd9>] lock_acquire+0x68/0x82
 [<c02aaf2a>] _spin_lock_bh+0x30/0x3d
 [<d886ad22>] sc92031_set_multicast_list+0x14/0x2d [sc92031]
 [<c025677e>] __dev_mc_upload+0x26/0x28
 [<c02568c4>] dev_mc_upload+0x25/0x3a
 [<c0254a02>] dev_open+0x4e/0x6e
 [<c025337f>] dev_change_flags+0x57/0x10b
 [<c028b237>] devinet_ioctl+0x235/0x546
 [<c028b837>] inet_ioctl+0x89/0xaa
 [<c0249cd1>] sock_ioctl+0x1ac/0x1ca
 [<c0170a70>] do_ioctl+0x1c/0x53
 [<c0170c93>] vfs_ioctl+0x1ec/0x203
 [<c0170cf3>] sys_ioctl+0x49/0x62
 [<c0103ca8>] sysenter_past_esp+0x5d/0x99
 =======================

[-mm patch] pcm_native: lockdep warning when launching jack

[Frederik Deweerdt]

On Mon, Apr 02, 2007 at 10:47:45PM -0700, Andrew Morton wrote:
> 
> ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/
> 
Hi,

When launching "jackd -d alsa", lockdep issues the following warning:
[39701.405086] =============================================
[39701.405093] [ INFO: possible recursive locking detected ]
[39701.405107] 2.6.21-rc5-mm4 #2
[39701.405109] ---------------------------------------------
[39701.405112] jackd/17366 is trying to acquire lock:
[39701.405114]  (&substream->self_group.lock){....}, at: [<c034d3c0>] snd_pcm_action_group+0x90/0x240
[39701.405131]
[39701.405131] but task is already holding lock:
[39701.405134]  (&substream->self_group.lock){....}, at: [<c034d63f>] snd_pcm_action_lock_irq+0x3f/0xb0
[39701.405141]
[39701.405142] other info that might help us debug this:
[39701.405145] 3 locks held by jackd/17366:
[39701.405147]  #0:  (snd_pcm_link_rwlock){....}, at: [<c034d627>] snd_pcm_action_lock_irq+0x27/0xb0
[39701.405155]  #1:  (&substream->group->lock){....}, at: [<c034d638>] snd_pcm_action_lock_irq+0x38/0xb0
[39701.405163]  #2:  (&substream->self_group.lock){....}, at: [<c034d63f>] snd_pcm_action_lock_irq+0x3f/0xb0
[39701.405171]
[39701.405171] stack backtrace:
[39701.405174]  [<c0103b8a>] show_trace_log_lvl+0x1a/0x30
[39701.405179]  [<c0104912>] show_trace+0x12/0x20
[39701.405183]  [<c01049c6>] dump_stack+0x16/0x20
[39701.405187]  [<c013b980>] __lock_acquire+0xbd0/0x1040
[39701.405193]  [<c013be60>] lock_acquire+0x70/0x90
[39701.405197]  [<c0407846>] _spin_lock+0x36/0x50
[39701.405203]  [<c034d3c0>] snd_pcm_action_group+0x90/0x240
[39701.405207]  [<c034d653>] snd_pcm_action_lock_irq+0x53/0xb0
[39701.405211]  [<c035046f>] snd_pcm_common_ioctl1+0x35f/0xfb0
[39701.405215]  [<c0351544>] snd_pcm_playback_ioctl1+0x34/0x420
[39701.405219]  [<c03519f3>] snd_pcm_playback_ioctl+0x43/0x50
[39701.405223]  [<c017ecc8>] do_ioctl+0x28/0x80
[39701.405229]  [<c017ed77>] vfs_ioctl+0x57/0x290
[39701.405233]  [<c017efe9>] sys_ioctl+0x39/0x60
[39701.405237]  [<c0102bf4>] sysenter_past_esp+0x5d/0x99
[39701.405240]  =======================

The attached lockdep annotation silences the warning.
Regards,
Frederik

Signed-off-by: Frederik Deweerdt <frederik.deweerdt@gmail.com>

--- 2.6.21-rc5-mm3/sound/core/pcm_native.c	2007-03-30 21:41:18.000000000 +0200
+++ 2.6.21-rc5-mm4/sound/core/pcm_native.c	2007-04-05 15:01:19.000000000 +0200
@@ -718,7 +718,8 @@ static int snd_pcm_action_group(struct a
 
 	snd_pcm_group_for_each_entry(s, substream) {
 		if (do_lock && s != substream)
-			spin_lock(&s->self_group.lock);
+			spin_lock_nested(&s->self_group.lock,
+					 SINGLE_DEPTH_NESTING);
 		res = ops->pre_action(s, state);
 		if (res < 0)
 			goto _unlock;

Re: [-mm patch] pcm_native: lockdep warning when launching jack

[Takashi Iwai]

At Thu, 5 Apr 2007 15:53:50 +0200,
Frederik Deweerdt wrote:
> 
> On Mon, Apr 02, 2007 at 10:47:45PM -0700, Andrew Morton wrote:
> > 
> > ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/
> > 
> Hi,
> 
> When launching "jackd -d alsa", lockdep issues the following warning:
> [39701.405086] =============================================
> [39701.405093] [ INFO: possible recursive locking detected ]
> [39701.405107] 2.6.21-rc5-mm4 #2
> [39701.405109] ---------------------------------------------
> [39701.405112] jackd/17366 is trying to acquire lock:
> [39701.405114]  (&substream->self_group.lock){....}, at: [<c034d3c0>] snd_pcm_action_group+0x90/0x240
> [39701.405131]
> [39701.405131] but task is already holding lock:
> [39701.405134]  (&substream->self_group.lock){....}, at: [<c034d63f>] snd_pcm_action_lock_irq+0x3f/0xb0
> [39701.405141]
> [39701.405142] other info that might help us debug this:
> [39701.405145] 3 locks held by jackd/17366:
> [39701.405147]  #0:  (snd_pcm_link_rwlock){....}, at: [<c034d627>] snd_pcm_action_lock_irq+0x27/0xb0
> [39701.405155]  #1:  (&substream->group->lock){....}, at: [<c034d638>] snd_pcm_action_lock_irq+0x38/0xb0
> [39701.405163]  #2:  (&substream->self_group.lock){....}, at: [<c034d63f>] snd_pcm_action_lock_irq+0x3f/0xb0
> [39701.405171]
> [39701.405171] stack backtrace:
> [39701.405174]  [<c0103b8a>] show_trace_log_lvl+0x1a/0x30
> [39701.405179]  [<c0104912>] show_trace+0x12/0x20
> [39701.405183]  [<c01049c6>] dump_stack+0x16/0x20
> [39701.405187]  [<c013b980>] __lock_acquire+0xbd0/0x1040
> [39701.405193]  [<c013be60>] lock_acquire+0x70/0x90
> [39701.405197]  [<c0407846>] _spin_lock+0x36/0x50
> [39701.405203]  [<c034d3c0>] snd_pcm_action_group+0x90/0x240
> [39701.405207]  [<c034d653>] snd_pcm_action_lock_irq+0x53/0xb0
> [39701.405211]  [<c035046f>] snd_pcm_common_ioctl1+0x35f/0xfb0
> [39701.405215]  [<c0351544>] snd_pcm_playback_ioctl1+0x34/0x420
> [39701.405219]  [<c03519f3>] snd_pcm_playback_ioctl+0x43/0x50
> [39701.405223]  [<c017ecc8>] do_ioctl+0x28/0x80
> [39701.405229]  [<c017ed77>] vfs_ioctl+0x57/0x290
> [39701.405233]  [<c017efe9>] sys_ioctl+0x39/0x60
> [39701.405237]  [<c0102bf4>] sysenter_past_esp+0x5d/0x99
> [39701.405240]  =======================
> 
> The attached lockdep annotation silences the warning.
> Regards,
> Frederik
> 
> Signed-off-by: Frederik Deweerdt <frederik.deweerdt@gmail.com>

Thanks, applied to ALSA tree now.

(I did a similar fix ago but forgot to merge to the upstream...)


Takashi

> 
> --- 2.6.21-rc5-mm3/sound/core/pcm_native.c	2007-03-30 21:41:18.000000000 +0200
> +++ 2.6.21-rc5-mm4/sound/core/pcm_native.c	2007-04-05 15:01:19.000000000 +0200
> @@ -718,7 +718,8 @@ static int snd_pcm_action_group(struct a
>  
>  	snd_pcm_group_for_each_entry(s, substream) {
>  		if (do_lock && s != substream)
> -			spin_lock(&s->self_group.lock);
> +			spin_lock_nested(&s->self_group.lock,
> +					 SINGLE_DEPTH_NESTING);
>  		res = ops->pre_action(s, state);
>  		if (res < 0)
>  			goto _unlock;
> 

RAID1 "out of memory" error, was Re: 2.6.21-rc5-mm4

[Reuben Farrelly]

Hi,

On 3/04/2007 3:47 PM, Andrew Morton wrote:
> ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/
> 
> - The oops in git-net.patch has been fixed, so that tree has been restored. 
>   It is huge.
> 
> - Added the device-mapper development tree to the -mm lineup (Alasdair
>   Kergon).  It is a quilt tree, living at
>   ftp://ftp.kernel.org/pub/linux/kernel/people/agk/patches/2.6/editing/.
> 
> - Added davidel's signalfd stuff.

Looks like some damage, or maybe intolerance to on-disk damage, to RAID-1.

md1 is the first array on the disk, and it refuses to start up on boot, or after 
boot.

tornado ~ # cat /proc/mdstat
Personalities : [raid1]
md1 : inactive sda1[0] sdc1[1]
       208640 blocks

md3 : active raid1 sdc3[1] sda3[0]
       20008832 blocks [2/2] [UU]
       bitmap: 0/153 pages [0KB], 64KB chunk

md5 : active raid1 sdc5[1] sda5[0]
       10008384 blocks [2/2] [UU]
       bitmap: 4/153 pages [16KB], 32KB chunk

md6 : active raid1 sdc6[1] sda6[0]
       10008384 blocks [2/2] [UU]
       bitmap: 0/153 pages [0KB], 32KB chunk

md8 : active raid1 sdc8[1] sda8[0]
       1003904 blocks [2/2] [UU]
       bitmap: 0/123 pages [0KB], 4KB chunk

md10 : active raid1 sdc10[1] sda10[0]
       119933120 blocks [2/2] [UU]
       bitmap: 1/229 pages [4KB], 256KB chunk

md2 : active raid1 sdc2[1] sda2[0]
       100004544 blocks [2/2] [UU]
       bitmap: 10/191 pages [40KB], 256KB chunk

unused devices: <none>
tornado ~ #

tornado ~ # mdadm --examine /dev/sda1
/dev/sda1:
           Magic : a92b4efc
         Version : 00.90.00
            UUID : f5c2e565:5ed956c0:33b08c07:16154426
   Creation Time : Fri Feb  2 10:16:29 2007
      Raid Level : raid1
   Used Dev Size : 104320 (101.89 MiB 106.82 MB)
      Array Size : 104320 (101.89 MiB 106.82 MB)
    Raid Devices : 2
   Total Devices : 2
Preferred Minor : 1

     Update Time : Fri Apr  6 02:06:17 2007
           State : clean
Internal Bitmap : present
  Active Devices : 2
Working Devices : 2
  Failed Devices : 0
   Spare Devices : 0
        Checksum : d3668aaa - correct
          Events : 0.368


       Number   Major   Minor   RaidDevice State
this     0       8        1        0      active sync   /dev/sda1

    0     0       8        1        0      active sync   /dev/sda1
    1     1       8       33        1      active sync   /dev/sdc1
tornado ~ # mdadm --examine /dev/sdc1
/dev/sdc1:
           Magic : a92b4efc
         Version : 00.90.00
            UUID : f5c2e565:5ed956c0:33b08c07:16154426
   Creation Time : Fri Feb  2 10:16:29 2007
      Raid Level : raid1
   Used Dev Size : 104320 (101.89 MiB 106.82 MB)
      Array Size : 104320 (101.89 MiB 106.82 MB)
    Raid Devices : 2
   Total Devices : 2
Preferred Minor : 1

     Update Time : Fri Apr  6 02:06:17 2007
           State : clean
Internal Bitmap : present
  Active Devices : 2
Working Devices : 2
  Failed Devices : 0
   Spare Devices : 0
        Checksum : d3668acc - correct
          Events : 0.368


       Number   Major   Minor   RaidDevice State
this     1       8       33        1      active sync   /dev/sdc1

    0     0       8        1        0      active sync   /dev/sda1
    1     1       8       33        1      active sync   /dev/sdc1
tornado ~ #


tornado ~ # mdadm --assemble /dev/md1 /dev/sda1 /dev/sdc1
mdadm: device /dev/md1 already active - cannot assemble it
tornado ~ # mdadm --run /dev/md1
mdadm: failed to run array /dev/md1: Cannot allocate memory
tornado ~ #

and looking at a dmesg, this is logged:

md: bind<sdc1>
md: bind<sda1>
raid1: raid set md1 active with 2 out of 2 mirrors
md1: bitmap initialized from disk: read 0/1 pages, set 0 bits, status: -12
md1: failed to create bitmap (-12)
md: pers->run() failed ...

tornado ~ # uname -a
Linux tornado 2.6.21-rc5-mm4 #1 SMP Thu Apr 5 23:47:42 EST 2007 x86_64 Intel(R) 
Pentium(R) 4 CPU 3.00GHz GenuineIntel GNU/Linux
tornado ~ #

The last known version that worked was 2.6.21-rc3-mm1 - I haven't been testing 
out the -mm releases so much lately.

Also, Andrew, can you please restart posting/cc'ing your -mm announcements to 
the linux-kernel-announce@vger.kernel.org list?  Seems this stopped around about 
2.6.20, it was handy.

.config is up at http://www.reub.net/files/kernel/configs/2.6.21-rc5-mm4

Thanks,
Reuben

Re: RAID1 "out of memory" error, was Re: 2.6.21-rc5-mm4

[Andrew Morton]

On Fri, 06 Apr 2007 02:33:03 +1000
Reuben Farrelly <reuben-linuxkernel@reub.net> wrote:

> Hi,
> 
> On 3/04/2007 3:47 PM, Andrew Morton wrote:
> > ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/
> > 
> > - The oops in git-net.patch has been fixed, so that tree has been restored. 
> >   It is huge.
> > 
> > - Added the device-mapper development tree to the -mm lineup (Alasdair
> >   Kergon).  It is a quilt tree, living at
> >   ftp://ftp.kernel.org/pub/linux/kernel/people/agk/patches/2.6/editing/.
> > 
> > - Added davidel's signalfd stuff.
> 
> Looks like some damage, or maybe intolerance to on-disk damage, to RAID-1.
> 
> md1 is the first array on the disk, and it refuses to start up on boot, or after 
> boot.
> 
> ...
> 
> tornado ~ # mdadm --assemble /dev/md1 /dev/sda1 /dev/sdc1
> mdadm: device /dev/md1 already active - cannot assemble it
> tornado ~ # mdadm --run /dev/md1
> mdadm: failed to run array /dev/md1: Cannot allocate memory
> tornado ~ #
> 
> and looking at a dmesg, this is logged:
> 
> md: bind<sdc1>
> md: bind<sda1>
> raid1: raid set md1 active with 2 out of 2 mirrors
> md1: bitmap initialized from disk: read 0/1 pages, set 0 bits, status: -12
> md1: failed to create bitmap (-12)
> md: pers->run() failed ...
> 
> tornado ~ # uname -a
> Linux tornado 2.6.21-rc5-mm4 #1 SMP Thu Apr 5 23:47:42 EST 2007 x86_64 Intel(R) 
> Pentium(R) 4 CPU 3.00GHz GenuineIntel GNU/Linux
> tornado ~ #
> 
> The last known version that worked was 2.6.21-rc3-mm1 - I haven't been testing 
> out the -mm releases so much lately.

OK.  I assume that bitmap->chunks in bitmap_init_from_disk() has some
unexpectedly large value.

I don't _think_ there's anything in -mm which would have triggered this. 
Does mainline do the same thing?

I guess it's possible that the code in git-md-accel.patch accidentally
broke things.  Perhaps try disabling CONFIG_DMA_ENGINE?

> Also, Andrew, can you please restart posting/cc'ing your -mm announcements to 
> the linux-kernel-announce@vger.kernel.org list?  Seems this stopped around about 
> 2.6.20, it was handy.

hm.  I always Bcc linux-kernel-announce@vger.kernel.org.  I assume that its
filters didn't get updated after the s/osdl/linux-foundation/ thing.  I'll
talk to people, thanks.

> .config is up at http://www.reub.net/files/kernel/configs/2.6.21-rc5-mm4
> 

Re: RAID1 "out of memory" error, was Re: 2.6.21-rc5-mm4

[Dan Williams]

On 4/5/07, Andrew Morton <akpm@linux-foundation.org> wrote:
> On Fri, 06 Apr 2007 02:33:03 +1000
> Reuben Farrelly <reuben-linuxkernel@reub.net> wrote:
>
> > Hi,
> >
> > On 3/04/2007 3:47 PM, Andrew Morton wrote:
> > > ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/
> > >
> > > - The oops in git-net.patch has been fixed, so that tree has been restored.
> > >   It is huge.
> > >
> > > - Added the device-mapper development tree to the -mm lineup (Alasdair
> > >   Kergon).  It is a quilt tree, living at
> > >   ftp://ftp.kernel.org/pub/linux/kernel/people/agk/patches/2.6/editing/.
> > >
> > > - Added davidel's signalfd stuff.
> >
> > Looks like some damage, or maybe intolerance to on-disk damage, to RAID-1.
> >
> > md1 is the first array on the disk, and it refuses to start up on boot, or after
> > boot.
> >
> > ...
> >
> > tornado ~ # mdadm --assemble /dev/md1 /dev/sda1 /dev/sdc1
> > mdadm: device /dev/md1 already active - cannot assemble it
> > tornado ~ # mdadm --run /dev/md1
> > mdadm: failed to run array /dev/md1: Cannot allocate memory
> > tornado ~ #
> >
> > and looking at a dmesg, this is logged:
> >
> > md: bind<sdc1>
> > md: bind<sda1>
> > raid1: raid set md1 active with 2 out of 2 mirrors
> > md1: bitmap initialized from disk: read 0/1 pages, set 0 bits, status: -12
> > md1: failed to create bitmap (-12)
> > md: pers->run() failed ...

Is this the dmesg from boot or the dmesg after running the mdadm --run command?

> >
> > tornado ~ # uname -a
> > Linux tornado 2.6.21-rc5-mm4 #1 SMP Thu Apr 5 23:47:42 EST 2007 x86_64 Intel(R)
> > Pentium(R) 4 CPU 3.00GHz GenuineIntel GNU/Linux
> > tornado ~ #
> >
> > The last known version that worked was 2.6.21-rc3-mm1 - I haven't been testing
> > out the -mm releases so much lately.
>
> OK.  I assume that bitmap->chunks in bitmap_init_from_disk() has some
> unexpectedly large value.
>
> I don't _think_ there's anything in -mm which would have triggered this.
> Does mainline do the same thing?
>
> I guess it's possible that the code in git-md-accel.patch accidentally
> broke things.  Perhaps try disabling CONFIG_DMA_ENGINE?
>

git-md-accel.patch does not touch anything in the raid1 path, but I
guess stranger things have happened.

--
Dan

Re: RAID1 "out of memory" error, was Re: 2.6.21-rc5-mm4

[Neil Brown]

On Friday April 6, reuben-linuxkernel@reub.net wrote:
> 
> Looks like some damage, or maybe intolerance to on-disk damage, to RAID-1.

Difference is that kzalloc(0, ) now returns NULL.  Maybe it is a
SLUB/SLAB difference? (So maybe it did use memory it shouldn't have
before, but now it fails, which is the better behaviour).

This patch fixes the maths and should probably go in various 'stable'
kernels.  Bug is in 2.6.18, but not 2.6.16.

Patch won't work for 2.6.18 as DIV_ROUND_UP is missing, but 2.6.19 and
later have it.

Thanks for the bug report.

NeilBrown


-----------------------------
Fix calculation for size of filemap_attr array in md/bitmap.

If 'num_pages' were ever 1 more than a multiple of 8 (32bit platforms)
for of 16 (64 bit platforms). filemap_attr would be allocated one
'unsigned long' shorter than required.  We need a round-up in there.


Signed-off-by: Neil Brown <neilb@suse.de>

### Diffstat output
 ./drivers/md/bitmap.c |    4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff .prev/drivers/md/bitmap.c ./drivers/md/bitmap.c
--- .prev/drivers/md/bitmap.c	2007-04-11 13:24:50.000000000 +1000
+++ ./drivers/md/bitmap.c	2007-04-11 13:24:59.000000000 +1000
@@ -863,9 +863,7 @@ static int bitmap_init_from_disk(struct 
 
 	/* We need 4 bits per page, rounded up to a multiple of sizeof(unsigned long) */
 	bitmap->filemap_attr = kzalloc(
-		(((num_pages*4/8)+sizeof(unsigned long)-1)
-		 /sizeof(unsigned long))
-		*sizeof(unsigned long),
+		roundup( DIV_ROUND_UP(num_pages*4, 8), sizeof(unsigned long)),
 		GFP_KERNEL);
 	if (!bitmap->filemap_attr)
 		goto out;

Re: 2.6.21-rc5-mm4

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1142 bytes --]

On Mon, 02 Apr 2007 22:47:45 PDT, Andrew Morton said:
> ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/

Am seeing an Oops 'cannot handle kernel paging request' during late
system startup, hand-copied traceback follows:

avc_has_perm_noaudit+0x2bf/0x506
avc_has_perm+0x2b/0x5b
selinux_socket_stream_connect+0x7e/0xc3
unix_stream_connect+0x202/0x3f3
sys_connect+0x7e/0xa4
tracesys+0xde/0xe1

I've not identified exactly when it happens, but it's towards the very end of
handling /etc/rc5.d, it's already up to the S98's.  Odd thing is it only happens
when I start with RedHat's 'graphical boot', and may be related to the shutdown
of the X server that's displaying the boot progress preparing to launch the
X server for gdm logins (as I'm also seeing a hang sometimes when shutting
down - so it is possibly a "shutting down X server nukes system" bug).

Figured I'd toss this heads-up in case it rings any bells, while I go do
the bisection dance on -rc5-mm4 (-mm2 is OK, and -mm3 doesn't boot for me
for other reasons I didn't chase down before -mm4 came out and fixed it, so
I have a ways to bisect)

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: 2.6.21-rc5-mm4

[Andrew Morton]

On Thu, 05 Apr 2007 13:02:59 -0400
Valdis.Kletnieks@vt.edu wrote:

> On Mon, 02 Apr 2007 22:47:45 PDT, Andrew Morton said:
> > ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.21-rc5/2.6.21-rc5-mm4/
> 
> Am seeing an Oops 'cannot handle kernel paging request' during late
> system startup, hand-copied traceback follows:
> 
> avc_has_perm_noaudit+0x2bf/0x506
> avc_has_perm+0x2b/0x5b
> selinux_socket_stream_connect+0x7e/0xc3
> unix_stream_connect+0x202/0x3f3
> sys_connect+0x7e/0xa4
> tracesys+0xde/0xe1
> 
> I've not identified exactly when it happens, but it's towards the very end of
> handling /etc/rc5.d, it's already up to the S98's.  Odd thing is it only happens
> when I start with RedHat's 'graphical boot', and may be related to the shutdown
> of the X server that's displaying the boot progress preparing to launch the
> X server for gdm logins (as I'm also seeing a hang sometimes when shutting
> down - so it is possibly a "shutting down X server nukes system" bug).

Thanks.

I'd have thought that the full trace could be captured with netconsole.

> Figured I'd toss this heads-up in case it rings any bells, while I go do
> the bisection dance on -rc5-mm4 (-mm2 is OK, and -mm3 doesn't boot for me
> for other reasons I didn't chase down before -mm4 came out and fixed it, so
> I have a ways to bisect)
> 

No, I'm not aware of anyone else hitting anything like that.

Bisection would be good, and probably pretty quick - I'd pick git-net.patch
as the first pivot point.

But we'd still be wanting the full trace if poss please.

Re: 2.6.21-rc5-mm4

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1213 bytes --]

On Thu, 05 Apr 2007 13:31:09 PDT, Andrew Morton said:
> On Thu, 05 Apr 2007 13:02:59 -0400, Valdis.Kletnieks@vt.edu wrote:

> > Am seeing an Oops 'cannot handle kernel paging request' during late
> > system startup, hand-copied traceback follows:
> > 
> > avc_has_perm_noaudit+0x2bf/0x506
> > avc_has_perm+0x2b/0x5b
> > selinux_socket_stream_connect+0x7e/0xc3
> > unix_stream_connect+0x202/0x3f3
> > sys_connect+0x7e/0xa4
> > tracesys+0xde/0xe1

> Thanks.
> 
> I'd have thought that the full trace could be captured with netconsole.

I didn't have a second box available at first.  Then I blew close to 45
minutes trying to figure out why netconsole was totally failing to work,
before I found this in .config:

# CONFIG_NETCONSOLE is not set

"Do'h!" -- H. Simpson

Unfortunately, defining netconsole caused NETPOLL to be defined, which caused
a recompile of half the known world, and the symptoms of the crash moved.

Film at 11, once I figure out what's going on, and fix the testbed in my
office so I can actually catch this sucker - I may have to string a serial
cable.  One solid good data point:

21-rc5 with only the -mm4 'origin.patch' applied is OK, so whatever the
issue is, it's not in Linus's tree.


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: 2.6.21-rc5-mm4

[Andrew Morton]

On Fri, 06 Apr 2007 11:26:24 -0400
Valdis.Kletnieks@vt.edu wrote:

> On Thu, 05 Apr 2007 13:31:09 PDT, Andrew Morton said:
> > On Thu, 05 Apr 2007 13:02:59 -0400, Valdis.Kletnieks@vt.edu wrote:
> 
> > > Am seeing an Oops 'cannot handle kernel paging request' during late
> > > system startup, hand-copied traceback follows:
> > > 
> > > avc_has_perm_noaudit+0x2bf/0x506
> > > avc_has_perm+0x2b/0x5b
> > > selinux_socket_stream_connect+0x7e/0xc3
> > > unix_stream_connect+0x202/0x3f3
> > > sys_connect+0x7e/0xa4
> > > tracesys+0xde/0xe1
> 
> > Thanks.
> > 
> > I'd have thought that the full trace could be captured with netconsole.
> 
> I didn't have a second box available at first.  Then I blew close to 45
> minutes trying to figure out why netconsole was totally failing to work,
> before I found this in .config:
> 
> # CONFIG_NETCONSOLE is not set
> 
> "Do'h!" -- H. Simpson
> 
> Unfortunately, defining netconsole caused NETPOLL to be defined, which caused
> a recompile of half the known world, and the symptoms of the crash moved.
> 
> Film at 11, once I figure out what's going on, and fix the testbed in my
> office so I can actually catch this sucker - I may have to string a serial
> cable.  One solid good data point:
> 
> 21-rc5 with only the -mm4 'origin.patch' applied is OK, so whatever the
> issue is, it's not in Linus's tree.
> 

Oh well.  If it's all too much fuss, feel free to send the .config.  If it
happens on my machine(s) I can bisect it real quick.