Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Mark_H_Johnson]

>do you have PREEMPT_REALTIME enabled? The above trace is a direct
>interrupt - only the timer interrupt is allowed to execute directly in
>the PREEMPT_REALTIME model - things break badly if it happens for any
>other interrupt (such as the soundcard IRQ).
Yes I have PREEMPT_REALTIME enabled.

The thing that comes to mind is I do have a script that does
  echo 0 > '/proc/irq/10/Esoniq AudioPCI/threaded
as part of ensuring the all the preemption stuff was set right. I may
have run that script prior to getting those messages. I thought you
said before that the non threaded IRQ stuff was disabled. Perhaps this
interface needs to be disabled as well [unless you really decide to
fix this limitation...].

I was already going into that script to add something like...
  for N in 1 3 4 6 8 10 11 12 14 15 ; do
    chrt -p -f 99 `pidof "IRQ $N"`
  done
to make all the threaded IRQ's max priority RT fifo tasks. I can
certainly comment out the IRQ thread disable code while I'm at it.

--Mark H Johnson
  <mailto:Mark_H_Johnson@raytheon.com>

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

* Mark_H_Johnson@raytheon.com <Mark_H_Johnson@raytheon.com> wrote:

> >do you have PREEMPT_REALTIME enabled? The above trace is a direct
> >interrupt - only the timer interrupt is allowed to execute directly in
> >the PREEMPT_REALTIME model - things break badly if it happens for any
> >other interrupt (such as the soundcard IRQ).
> Yes I have PREEMPT_REALTIME enabled.
> 
> The thing that comes to mind is I do have a script that does
>   echo 0 > '/proc/irq/10/Esoniq AudioPCI/threaded
>
> as part of ensuring the all the preemption stuff was set right. I may
> have run that script prior to getting those messages. I thought you
> said before that the non threaded IRQ stuff was disabled. Perhaps this
> interface needs to be disabled as well [unless you really decide to
> fix this limitation...].

argh, there was a typo in that change so the 'threaded' flags werent
really disabled. So i ended up only disabling the global hardirq_preempt
flag but not the per-handler 'threaded' flags. Ouch!

I've uploaded the -U9.1 patch that has the fix, does it work any better
than previous kernels?

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Mark_H_Johnson]

With a workaround patch for the boot time BUG, I was able to get ...
 - to single user mode w/o any errors
 - a [NOW] non fatal error getting the network up (telinit 3)
 - no further errors getting the X server up (telinit 5)
 - able to hear sample audio
 - system stayed up all night (daemons were stable...)
This is the first time in about two weeks that I had a reasonably stable
system (last known good is -T3).

I was about to run my normal stress tests when the system locked up.

The symptom was the display stopped updating / no mouse motion. Apparently
caused while I was dragging a window with the mouse (USB). We may still
have problems in that area. No apparent response to Alt-Sysrq keys;
hardware reset was sufficient to reboot.

Will check the system logs to see what I can find.

--Mark H Johnson
  <mailto:Mark_H_Johnson@raytheon.com>

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

* Mark_H_Johnson@raytheon.com <Mark_H_Johnson@raytheon.com> wrote:

> I was about to run my normal stress tests when the system locked up.
> 
> The symptom was the display stopped updating / no mouse motion.
> Apparently caused while I was dragging a window with the mouse (USB).
> We may still have problems in that area. No apparent response to
> Alt-Sysrq keys; hardware reset was sufficient to reboot.

the soundcard IRQ trace you got is interesting:

 BUG: sleeping function called from invalid context X(2948) at kernel/mutex.c:25 Oct 21 07:53:02 localhost kernel: in_atomic():1 [00010000], irqs_disabled():0
  [<c011f06a>] __might_sleep+0xca/0xe0 (12)
  [<c01387e6>] _mutex_lock+0x26/0x50 (36)
  [<e0a549b6>] snd_audiopci_interrupt+0x46/0xf0 [snd_ens1371] (20)
  [<c01436f6>] handle_IRQ_event+0x46/0x80 (24)
  [<c0143837>] __do_IRQ+0x107/0x160 (32)
  [<c010a299>] do_IRQ+0x59/0x90 (36)
  [<c0108510>] common_interrupt+0x18/0x20 (20)
 preempt count: 00010001

do you have PREEMPT_REALTIME enabled? The above trace is a direct
interrupt - only the timer interrupt is allowed to execute directly in
the PREEMPT_REALTIME model - things break badly if it happens for any
other interrupt (such as the soundcard IRQ).

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> the soundcard IRQ trace you got is interesting:
> 
>  BUG: sleeping function called from invalid context X(2948) at kernel/mutex.c:25 Oct 21 07:53:02 localhost kernel: in_atomic():1 [00010000], irqs_disabled():0
>   [<c011f06a>] __might_sleep+0xca/0xe0 (12)
>   [<c01387e6>] _mutex_lock+0x26/0x50 (36)
>   [<e0a549b6>] snd_audiopci_interrupt+0x46/0xf0 [snd_ens1371] (20)
>   [<c01436f6>] handle_IRQ_event+0x46/0x80 (24)
>   [<c0143837>] __do_IRQ+0x107/0x160 (32)
>   [<c010a299>] do_IRQ+0x59/0x90 (36)
>   [<c0108510>] common_interrupt+0x18/0x20 (20)
>  preempt count: 00010001
> 
> do you have PREEMPT_REALTIME enabled? The above trace is a direct
> interrupt - only the timer interrupt is allowed to execute directly in
> the PREEMPT_REALTIME model - things break badly if it happens for any
> other interrupt (such as the soundcard IRQ).

this also seems to be the major cause of the other problems in your log:
kernel preempting off a hardirq context and then confusing other kernel
code. It's surprising that it booted up at all! Now how did the
soundcard IRQ end up being non-threaded?

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Mark_H_Johnson]

>i have released the -U8 Real-Time Preemption patch:
>
>
http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U8

First attempt at booting this version, made it to single user OK but
telinit 3 stopped while starting up the network with the following messages
on console...

INIT: Switching to runlevel: 3
INIT: Sending processes the TERM signal
INIT: Sending processes the KILL signal
Applying Intel IA32 Microcode update:   [ OK ]
Checking for new hardware ----------------[ cut here ]---------------
kernel BUG at include/asm/atomic.h:135!
Invalid operand: 0000 [#1]
PREEMPT SMP
Modules linked in: 8139too mii floppy sg scsi_mod microcode dm_mod uhci_hcd
ext3 jbd
CPU:    0
EIP:    0060:[<c02b8426>]    Not tainted VLI
EFLAGS: 00010246   (2.6.9-rc4-mm1-RT-U8)
EIP is at qdisc_destroy+0x76/0x80
eax: 00000000   ebx: dedf9000   ecx: c037ede0   edx: c037ede0
esi: dedf9000   edi: c03747a8   ebp: df3d9e68   esp: df3d9e64
ds: 007b   es: 007b   ss: 0068   preempt: 00000001
Process modprobe (pid: 1594, threadinfo=df3d8000 task=db708690)
Stack: dedf9000 df3d9e84 c02b863d c037ede0 df3d9e74 df3d9e74 c03423e4
dedf9000
       df3d9ea8 c02a9bdb dedf9000 c01148b0 e0836b10 dedf9420 dedf9000
e083b148
       0807a804 df3d9eb8 c024192e dedf9000 dedf9000 df3d9ed8 e0836b3a
dedf9000
Call Trace:
 [<c02b863d>] dev_shutdown+0x3d/0xa0 (12)
 [<c02a9bdb>] unregister_netdevice+0x13b/0x280 (28)
 [<c01148b0>] mcount+0x14/0x18 (8)
 [<e0836b10>] rtl8139_remove_one+0x0/0xa0 [8139too] (4)
 [<c024192e>] unregister_netdev+0x1e/0x30 (24)
 [<e0836b3a>] rtl8139_remove_one+0x2a/0xa0 [8139too] (16)
 [<c01148b0>] mcount+0x14/0x18 (12)
 [<c01e3a06>] pci_device_remove+0x76/0x80 (20)
 [<c02300bb>] device_detach_shutdown+0xb/0x40 (12)
 [<c022d207>] device_release_driver+0x67/0x70 (12)
 [<c022d23b>] device_detach+0x2b/0x40 (24)
 [<c022d6af>] bus_remove_driver+0x3f/0x70 (20)
 [<c022dbb9>] driver_unregister+0x19/0x30 (20)
 [<c01e3cac>] pci_unregister_driver+0x1c/0x30 (16)
 [<e0838ae7>] rtl8139_cleanup_module+0x17/0x1b [8139too] (16)
 [<c013c751>] sys_delete_module+0x121/0x150 (12)
 [<c01594f4>] sys_munmap+0x54/0x70 (64)
 [<c0118560>] do_page_fault+0x0/0x6d0 (16)
 [<c0107b49>] sysenter_past_esp+0x52/0x71 (16)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x1f/0x80 / (die+0x44/0x190)
.. entry 2: print_traces_0x1d/0x60 / (show_stack+0x8f/0xb0)

Code: 00 ba c0 82 2b c0 c7 43 04 00 02 20 00 5b 5d e9 81 bf e7 ff 0f 0b a5
00 e9
 4d 32 c0 eb cd 0f 0b a4 00 e9 4d 32 c0 eb b8 5b 5d c3 <0f> 0b 87 00 0d 4e
32 c0
 eb 93 55 89 e5 83 ec 10 89 5d f8 89 75

Updating /etc/fstab             [ OK ]
Setting network parameters:     [ OK ]
Bringing up loopback interface:

(no further console output at this point)

Alt-Sysrq-L showed active tasks were swapper (CPU 1) and IRQ 1 (CPU 0) at
cpu_idle and nmi_show_all_regs respectively. Able to repeat this more than
once with the same results each time. Tasks listed by Alt-Sysrq-T included
IRQ 6, S10network, minlogd, initlog, ifup, ip, and grep (all I could see
on the scrollback).

Synced disks (Alt-Sysrq-S) and rebooted (Alt-Sysrq-B) to try again. This
time
turning on syslog before telinit 3. Same results. Will send whatever showed
up
on disk separately.

--Mark H Johnson
  <mailto:Mark_H_Johnson@raytheon.com>

[patch] VP-2.6.9-rc4-mm1-T8

[Ingo Molnar]

i've uploaded the -T8 VP patch:

  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc4-mm1-T8

lots of stabilization of CONFIG_PREEMPT_REALTIME. It's still in
experimental status but general stability is improving.

Changes since -T7:

 - fixed a nasty category of bugs that were introduced by the use of 
   rwsems for rwlocks. rwsems are not read-recursive, while rwlocks are. 
   Fortunately it was not too hard to identify & fix recursive users of
   tasklist_lock, in fact each of these also qualifies as a cleanup. The 
   symptom of this bug was a soft-deadlocking of the system. 

 - fixed profiler locks, i believe this could resolve the bootup crash
   reported by K.R. Foley.

 - fixed VP / XFS namespace collision reported by Mark H. Johnson

 - fix one more final detail of the new zombie-task handling code

 - fixed 3c59x.c, fasync-handling, ipv6, module-loader runtime
   warnings reported by K.R. Foley.

 - fixed the ali5451 locking

to create a -T8 tree from scratch the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.8.tar.bz2
 + http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.9-rc4.bz2
 + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc4/2.6.9-rc4-mm1/2.6.9-rc4-mm1.bz2
 + http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc4-mm1-T8

	Ingo

[patch] VP-2.6.9-rc4-mm1-T9

[Ingo Molnar]

i've uploaded the -T9 VP patch:

  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc4-mm1-T9

this is a bugfixes-only release that should fix the highmem-related
issues reported by K.R. Foley and Mark H. Johnson: 3 more locks had to
be converted to raw.

to create a -T9 tree from scratch the patching order is:
 
   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.8.tar.bz2
 + http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.9-rc4.bz2
 + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc4/2.6.9-rc4-mm1/2.6.9-rc4-mm1.bz2
 + http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc4-mm1-T9

	Ingo

[patch] Real-Time Preemption, -VP-2.6.9-rc4-mm1-U0

[Ingo Molnar]

i'm pleased to announce a significantly improved version of the
Real-Time Preemption (PREEMPT_REALTIME) feature that i have been working
towards in the past couple of weeks.

the patch (against 2.6.9-rc4-mm1) can be downloaded from:

  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc4-mm1-U0

this is i believe the first correct conversion of the Linux kernel to a
fully preemptible (fully mutex-based) preemption model, while still
keeping all locking properties of Linux.

I also think that this feature can and should be integrated into the
upstream kernel sometime in the future. It will need improvements and
fixes and lots of testing, but i believe the basic concept is sound and
inclusion is manageable and desirable. So comments, testing and feedback
is more than welcome!

to recap the conceptual issues that needed solving: the previous patch
already converted a fair portion of spinlocks to mutexes, but that in a
fully preemptible kernel model the following locking primitives are
especially problematic:

	- RCU locking
	- per-cpu counters and per-cpu variables
	- tlb gather/release logic
	- seqlocks
	- atomic-kmaps
	- pte mapping

note that while we tend to think about these as SMP locking primitives,
in a fully preemptible model these locking rules are necessary for
correctness, even on a single-CPU embedded box.

Unfortunately none of the existing preemption patches solve these
problems: they concentrate on UP but these locking rules must not be
ignored on UP either!

(Bill Huey's mutex patch he just released is the one notable exception i
know about, which is also a correct implementation i believe, but it
doesnt attack these locking rules [yet]. Bill's locking hierarchy is i
believe quite similar to the -T9 patch - this i believe is roughly the
best one can get when only using spinlocks as a vehicle.)

In the previous (-T9) version of the Real-Time Preemption patch, the
above locking primitives kept large portions of kernel code
non-preemptable, causing a 'spreadout' of raw spinlocks and
non-preemptible sections to various kernel subsystems.

Another, even more problematic effect was that both network drivers and
block IO drivers got 'contaminated' by raw spinlocks, triggering lots of
per-driver changes and thus limiting testability. It is basically an
opt-in model to correctness which is bad from a maintainance and
upstream acceptance point of view. The -T9 patch was a prime example of
the problems the Linux locking primitives cause in a fully preemptible
kernel model.


To solve all these fundamental problems, i improved/fixed/changed all of
these locking methods to be preemption-friendly. Most of the time it was
necessary to introduce an additional API variant because e.g.
rcu_read_lock() is anonymous (it doesnt identify the data protected), so
i introduced a variant that takes the write-lock as an argument. In the
PREEMPT_REALTIME case we can thus properly serialize on that lock.

For per-cpu variables i introduced a new API variant that creates a
spinlock-array for the per-cpu-variable, and users must make sure the
cpu field doesnt change. Migration to another CPU can happen within the
critical section, but 'statistically' the variable is still per-CPU and
update correctness is fully preserved.

TLB shootdown was the source of another nasty type of critical section:
it keeps preemption disabled during much of the pagetable zapping and
also relies on a per-CPU variable to keep TLB state. The fix for
PREEMPT_REALTIME (on x86) was to implement a simpler but preemptible TLB
flushing method. This necessiated two minor API changes to the generic
TLB shootdown code - pretty straightforward ones.

Atomic kmaps were another source of non-preemptible sections:
pte_map/pte_unmap both used nontrivial functions within and ran for a
long time, creating a lock dependency and a latency problem as well. I
wrapped them via normal kmaps, which thus become preemptible. (The main
reason for atomic kmaps were non-preemptability constraints - but those
are not present in a fully preemptible model.)

seqlocks (used within the VFS and other places) were another problem:
the are now preemptible by default, the same auto-type-detection logic
applies to them as to preemptible/raw spinlocks: switching between a
preemptible and a non-preemptible type is done by changing the
prototype, the locking APIs stay the same.

The improvements to locking allowed the gradual 'pulling out' of all
raw-spinlocks from various kernel subsystems. In the -U0 patch i have
almost completely finished this 'pullout', and as a result the following
kernel subsystems are now completely spinlock-free and 100% mutex-based:

	- networking
	- IO subsystem
	- VFS and lowlevel filesystems
	- memory management (except the SLAB allocator lock)
	- signal code
	- all of drivers/* and sound/*

note: it is important that when PREEMPT_REALTIME is disabled, the old
locking rules apply and there is no performance impact whatsoever. So
what this implements is in essence a compile-time multi-tier locking
architecture enabling 4 types of preemption models:

  - stock                      (casual voluntary kernel preemption)
  - CONFIG_PREEMPT_VOLUNTARY   (lots of cond_resched() points)
  - CONFIG_PREEMPT             (involuntary preemption plus spinlocks)
  - CONFIG_PREEMPT_REALTIME    (everything is a mutex)

these models cover all the variants people are interested in: servers
with almost no latency worries, desktops with ~1msec needs and hard-RT
applications needing both microsecond-range latencies and provable
maximum latencies.

to quantitatively see the effects of these changes to the locking
landscape, here's the output of a script that disassembles the kernel
image and counts the number of actual spin lock/unlock function calls
done versus mutex lock/unlocks:

With PREEMPT_REALTIME disabled, all locks are spinlocks and old
semaphores:

        spinlock API calls:      5359  (71.6%)
        | old mutex API calls:   2120  (28.3%)
        | new mutex API calls:      2  (0%)
        all mutex API calls:     2122  (28.3%)
        --------------------------------------
        lock API calls:          7481 (100.0%)

with the -T9 kernel, which had the 'spread out' locking model, a
considerable portion of spinlocks were replaced by mutexes, but more
than 20% of usage was still spinlocks:

        spinlock API calls:      1835  (23.1%)
        | old mutex API calls:   2142  (26.9%)
        | new mutex API calls:   3961  (49.8%)
        all mutex API calls:     6103  (76.8%)
        --------------------------------------
        lock API calls:          7938 (100.0%)

here are some fresh numbers from Bill Huey's mmLinux kernel:

        spinlock API calls:      2452  (30.3%)
        all mutex API calls:     5614  (69.6%)
        --------------------------------------
        lock API calls:          8066 (100.0%)

(his mutex implementation directly falls back to up()/down() so the new
mutexes become part of the old mutexes.)

while i believe that the locking design is fundamentally incomplete in
the MontaVista kernel and thus is not directly comparable to
PREEMPT_REALTIME nor the mmLinux kernel, here are the stats from it
using a similar .config:

        spinlock API calls:      1444  (26.1%)
        | old mutex API calls:   2095  (37.9%)
        | new mutex API calls:   1981  (35.8%)
        all mutex API calls:     4076  (73.8%)
        --------------------------------------
        lock API calls:          5520 (100.0%)

(here it is visible that apparently a significant amount of [i believe
necessary] locking is missing from this kernel.)

finally, here are the stats from the new PREEMPT_REALTIME -U0 kernel:

        spinlock API calls:       491  (6.0%)
        | old mutex API calls:   2142  (26.2%)
        | new mutex API calls:   5536  (67.7%)
        all mutex API calls:     7678  (93.9%)
        --------------------------------------
        lock API calls:          8169 (100.0%)

note that almost all of the remaining spinlocks are held for a short
amount of time and have a finite maximum duration. They involve hardware
access, scheduling and interrupt handling and timers - almost all of
that code has O(1) characteristics.

what this means is that we are approaching hard-real-time domains ... 
using what is in essence the stock Linux kernel!

note that priority inheritance is still not part of this patch, but that
effort can now be centralized to the two basic Linux semaphore types,
solving the full spectrum of priority inheritance problems!

the code is still x86-only but only for practical reasons - other
architectures will be covered in the future as well.

to create a -U0 tree from scratch the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.8.tar.bz2
 + http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.9-rc4.bz2
 + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc4/2.6.9-rc4-mm1/2.6.9-rc4-mm1.bz2
 + http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc4-mm1-U0

	Ingo

[patch] Real-Time Preemption, -VP-2.6.9-rc4-mm1-U1

[Ingo Molnar]

i have released the -U1 PREEMPT_REALTIME patch:
 
  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc4-mm1-U1

this is a strict bugfixes-only release. With -U1 i cannot reproduce any
of the bugs on my testsystems anymore, but take care nevertheless, this
is still experimental code.

Changes since -U0:

 - bugfix: fixed the highmem related crash reported by Adam Heath and i 
   think this could also fix the crash reported by Mark H Johnson.

 - bugfix: fixed a number of networking related soft-lockups, caused by
   a deadlock scenarios in the ipv4, netfilter and net-xmit locking
   code. This could fix the lockup reported by Lorenzo Allegrucci.

 - bugfix: enable interrupts in the int3 handler - gdb will otherwise
   trigger a kernel debug message.

 - cleanup: reworked the RCU API wrappers, we now have the following
   variants:
 
     rcu_read_[un]lock_spin(&spinlock)
     rcu_read_[un]lock_bh_spin(&spinlock)
     rcu_read_[un]lock_sem(&semaphore)

   this change was necessary for the network locking fixes.

 - debugging helper: SysRq-T will now print the stacktrace of currently
   running tasks too. (They might be a bit unreliable occasionally but
   very useful to debug deadlocks.)

 - configurability fix: disabled the /proc/kernel/softirq_preemption and
   hardirq_preemption runtime flags (and the softirq-preempt= and
   hardirq-preempt= boot flags) if PREEMPT_REALTIME is enabled - in the
   fully preemptible model these must always be on.

there are no known bugs at this moment, so please re-report any issues 
you might still encounter.

to create a -U1 tree from scratch the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.8.tar.bz2
 + http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.9-rc4.bz2
 + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc4/2.6.9-rc4-mm1/2.6.9-rc4-mm1.bz2
 + http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc4-mm1-U1

	Ingo

[patch] Real-Time Preemption, -VP-2.6.9-rc4-mm1-U2

[Ingo Molnar]

i have released the -U2 PREEMPT_REALTIME patch:

  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc4-mm1-U2

this too is a bugfixes-only release, and it is still experimental code.

Changes since -U1:

 - bugfix: fix page_lock_anon_vma() crash reported by Adam Heath and 
   Lorenzo Allegrucci.

 - bugfix: fix selinux atomic-schedule warning messages, reported by
   Mark H Johnson.

 - bugfix: ip_tables atomic-schedule fix, fixes the messages reported by 
   Daniel Walker and K.R. Foley.

 - bugfix: fix warnings/deadlocks in inet_create(), reported by Mark H 
   Johnson.

 - bugfix: fixed a crash-in-shmfs-during-heavy-swapout bug

 - bugfix: enable preemption while doing mmdrop() in the scheduler - it 
   may schedule.

 - debugging feature: when PREEMPT_TIMING is enabled then the code also 
   keeps a trace/stack of preemption enabler EIPs. (if LATENCY_TRACE is 
   enabled as well then the parent EIP is recorded as well.) Whenever a 
   stack trace due to atomicity violations is printed, the preemption
   stack is printed as well. This makes it much easier to identify the 
   place that did the illegal preemption-disabling.

to create a -U2 tree from scratch the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.8.tar.bz2
 + http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.9-rc4.bz2
 + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc4/2.6.9-rc4-mm1/2.6.9-rc4-mm1.bz2
 + http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc4-mm1-U2

	Ingo

[patch] Real-Time Preemption, -VP-2.6.9-rc4-mm1-U3

[Ingo Molnar]

i have released the -U3 PREEMPT_REALTIME patch:

  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc4-mm1-U3

this is a buildfixes-only release, and it is still experimental code.

Changes since -U2:

 - build fix: fixes the latency.c compilation error reported by Adam 
   Heath.

 - build fix: fixes !HIGHMEM compilation, patch from Andrew Rodland

to create a -U3 tree from scratch the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.8.tar.bz2
 + http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.9-rc4.bz2
 + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc4/2.6.9-rc4-mm1/2.6.9-rc4-mm1.bz2
 + http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc4-mm1-U3

	Ingo

[patch] Real-Time Preemption, -VP-2.6.9-rc4-mm1-U4

[Ingo Molnar]

i have released the -U4 PREEMPT_REALTIME patch:

   http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc4-mm1-U4

this is a fixes-only release, and it is still experimental code.

Changes since -U3:

 - crash fix: fix the rtc_lock related crash reported by Bill Huey, the 
   rtc_lock is now a raw spinlock again.

 - crash fix: avoid recursion into timer code when PREEMPT_TIMING is 
   enabled.

 - crash/printout fix: revert some of selinux's locks back to raw 
   spinlocks. This could fix the problems reported by Mark H. Johnson, 
   Adam Heath.

 - build fix: fix the compilation problems reported by Lee Revell

 - debug feature: implemented 'print backtrace on all CPUs' on SMP 
   systems, SysRq+L will trigger it.

 - build cleanup: restructure the debug config options. This should 
   resolve the build problems and incompatible debug-options
   problems reported.

 - cleanup: move definitions around, turn on generic rwlocks instead of
   the x86-specific version.

i think all bugs that were reported with logs are fixed in -U4. Please
re-report any issue that might remain.

to create a -U4 tree from scratch the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.8.tar.bz2
 + http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.9-rc4.bz2
 + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc4/2.6.9-rc4-mm1/2.6.9-rc4-mm1.bz2
 + http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc4-mm1-U4

	Ingo

[patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U5

[Ingo Molnar]

i have released the -U5 Real-Time Preemption patch:

  http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc4-mm1-U5

this is a release intended to increase stability, but since it also
includes new debug features and related cleanups it might introduce new
regressions. Be careful.

there are two big changes:

 - debug feature: automatic semaphore/rwsem deadlock detection, based on
   the code from Igor Manyilov and Bill Huey.

this is a very nice feature that should help in debugging the remaining
deadlocks. The deadlock detection feature has already helped me fix a
bug that was causing hangs in the VFS, so it's really useful.

 - generic semaphore implementation

the generic semaphore implementation (which uses rwsems) makes it
possible to use the deadlock detection mechanism for all the mutex types
we currently have: semaphores, rw-semaphores, spinlock-mutexes and
rwlock-mutexes. Another benefit is that PREEMPT_REALTIME becomes much
more portable this way. (although it's still x86-only at the moment.)

other changes since -U4:

 - crash fix: fixed a possible "unbound recursion upon IRQ entry" bug.
   introduced preempt_schedule_irq() which now schedules without
   enabling interrupts again, preventing new IRQs from hitting this
   task again and triggering preemption. This might fix the
   'infinite stackdumps' problem Rui Nuno Capela was seeing.

 - deadlock fix: is_subdir()'s PREEMPT_REALTIME locking was buggy. This 
   could perhaps fix the other problem reported by Rui Nuno Capela.

 - i8253_lock fixes: apm, hd.c, gameport.c and analog.c were all 
   improperly importing the variable while overriding the prototype. 
   This fixes the bug reported by Florian Schmidt.

 - possible crash fix: one particular lock in selinux has to be 
   mutex-based, because while held it calls other mutex-using code.

 - two more selinux locks converted to raw spinlocks, because they were
   called from within raw-critical sections.

 - debug feature: enforce interrupts-enabled upon schedule().
   (Note that this does not break sleep_on() because sleep_on() does not
   disable interrupts in the PREEMPT_REALTIME mode. It might break with 
   !PREEMPT_REALTIME though.)

 - locking cleanup: converted the IPC code from raw spinlocks & RCU to
   spinlock-mutexes.

 - code cleanup: cleaned up the generic rwsem code.

 - debug feature: implemented /proc/sys/kernel/trace_verbose runtime
   flag (default:0), which enables a much more verbose printout in
   /proc/latency_trace.  This trace format can be useful in e.g. 
   debugging timestamp weirdnesses.

 - irqs-off fix: there was one codepath where irqd would call schedule() 
   with interrupts disabled.

 - debug feature: the NMI entries in the latency trace now also include 
   the last-observed-EFLAGS value. Can be useful in figuring out what a
   certain CPU is doing and why.

 - cleanup: fixed preemption-off ordering: often the spinlock (and 
   scheduler) code would re-enable preemption and interrupts in the
   wrong order, opening up a small window for an interrupt handler to
   fit in and increase the latency of that almost-finished critical
   section.

 - cleanup: consolidated various bug-printouts. It should now be easy to
   find whether anything bad happens even amongst lots of preempt-timing
   printouts: 'dmesg | grep BUG'.

to create a -U5 tree from scratch, the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.8.tar.bz2
 + http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.9-rc4.bz2
 + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc4/2.6.9-rc4-mm1/2.6.9-rc4-mm1.bz2
 + http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc4-mm1-U5

	Ingo

[patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U6

[Ingo Molnar]

i have released the -U6 Real-Time Preemption patch:
 
  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U6

this is a fixes-only release.

found and fixed the 'big bug' that was probably the one causing
stability problems for a number of people. There was a small window for
a task double-free race to occur, causing all sorts of crashes later on.
This bug could trigger on UP and SMP systems alike, on SMP being a bit
more frequent.

Also, a common networking deadlock was found and fixed as well, using
the deadlock detector.

Changes since -U5:

- crash bug: fix task double free caused by irq-preemption of 
  do_exit(). This got introduced in -U5 as part of a simplification of
  the zombie-reaping rewrite that the -U series did. That rewrite had an 
  unrobustness which got triggered by -U5 in a subtle way, opening up a
  small window at the end of do_exit() for an interrupt-triggered
  preemption to cause a double-free. This could fix some of the crashes
  reported by Rui Nuno Capela, Mark H Johnson.

- deadlock bug: fix networking deadlock reported by Matthew L Foster.
  Restructured the way the RT-RCU locking of ptype_lock is done - it's
  cleaner and more obvious now (besides being correct). This could also
  fix the deadlock reported by Michal Schmidt.

- deadlock bug: fix NFS startup breakage related to semaphore abuse,
  patch from Thomas Gleixner.

- build bug: fix aha152x.c, based on patch from K.R. Foley.

- build bug: fix compilation error in qla2xxx. (reported by Fernando
  Pablo Lopez-Lezcano and Mark H Johnson)

- build bug: fix !PREEMPT_REALTIME compilation error. (reported by
  Matthew L Foster)

- build bug: fix ipmi-watchdog compilation error. (reported by Mark H 
  Johnson)

- tracer fix: if an assert happens within the tracer then we'd get into
  infinite recursion. The fix was to correctly nest tracing on/off
  points.

- debug enhancement: added a few more asserts to catch underflowing 
  atomic counters. (this made the task double-free trigger earlier.)

- debug enhancement: extended CONFIG_DEBUG_STACKOVERFLOW to be 
  mcount()-driven as well. This helps in catching stack overflows much
  more reliably than the do_IRQ() based method.

to create a -U6 tree from scratch, the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.8.tar.bz2
 + http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.9-rc4.bz2
 + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc4/2.6.9-rc4-mm1/2.6.9-rc4-mm1.bz2
 + http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U6

	Ingo

[patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U7

[Ingo Molnar]

i have released the -U7 Real-Time Preemption patch:

  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U7

this too is a fixes-only release.

Changes since -U6:

 - crash fix: turn off 4K stacks when using RWSEM_DEADLOCK_DETECTION, 
   and tune down the default max # of tasks traced per semaphore. This 
   increases process-stack size and reduces the footprint of lock 
   objects. This should fix the bootup crash reported by Rui Nuno 
   Capela.

 - assert fix: fixed an ide-taskfile scheduling-with-irqs-off assert
   that Rui's .config triggers.

 - assert workaround: disabled PARPORT_1284 for now, this should fix the
   assert seen by Mark H Johnson.

 - NFS fix: clnt.c fix from Thomas Gleixner

 - debugging helper: print stackframe-size in backtraces.

 - large-stackframe fix: inflate.c fix

to create a -U7 tree from scratch, the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.8.tar.bz2
 + http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.9-rc4.bz2
 + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc4/2.6.9-rc4-mm1/2.6.9-rc4-mm1.bz2
 + http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U7

	Ingo

[patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

i have released the -U8 Real-Time Preemption patch:

  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U8

this too is a fixes-only release. It includes the many semaphore-abuse
and sleep_on() fixes/improvements from Thomas Gleixner, and it also
includes a couple of semaphore related fixes.

I believe the semaphore fixes should resolve a number of the deadlocks
reported for -U7.

In particular it seems the only sane and reliable way to convert RCU
locking was to allow the following semantics for rwsems: allow reads to
nest, and allow self-read-recursion of a self-write-held semaphore. My
current implementation for this allows semaphore unfairness, but that
can be fixed later on. Most importantly, the RCU to RT-locking
conversions are much more automatic now and map nicely to what the code
is doing upstream. Most of the time they involve a conversion of a
spinlock or semaphore into a rwlock or rwsem. The old code maps to new
code almost automatically, the only manual work needed was to associate
the rcu_read_lock() with the writers-lock that it excludes against,
which is a pretty clear (but not automatic, and hence not automatable)
decision. This way i could convert some more networking code, and
simplify the older changes and hopefully get rid of some deadlocks. The
locking API is still not in its final form, but it's getting closer.

Changes since -U7:

 - deadlock fix: sysfs/driver-base semaphore fixes from Thomas Gleixner

 - deadlock fix: scsi semaphore fixes from Thomas Gleixner

 - NFS sleep_on() fixes from Thomas Gleixner

 - rawmidid.c sleep_on() fix from Thomas Gleixner

 - [ i've added more wait_for_completion_*() primitives, to ease 
     conversion of other semaphore-(ab-)using code. ]

 - make rwsems self-recursive

 - RCU lock conversion: convert rtnl_sem RCU use.

 - netfilter deadlock fix - clean up RCU locking.

to create a -U8 tree from scratch, the patching order is:

   http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.8.tar.bz2
 + http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.9-rc4.bz2
 + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc4/2.6.9-rc4-mm1/2.6.9-rc4-mm1.bz2
 + http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U8

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

> Changes since -U7:
> 

- fix block-loopback assert reported by Mark H Johnson, Matthew L
  Foster and Rui Nuno Capela. (usually triggers during 'make install' 
  of a kernel compile.)

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Rui Nuno Capela]

[-- Attachment #1: Type: text/plain, Size: 2852 bytes --]

Ingo Molnar wrote:
>
>> Changes since -U7:
>>
>
> - fix block-loopback assert reported by Mark H Johnson, Matthew L
>   Foster and Rui Nuno Capela. (usually triggers during 'make install'
>   of a kernel compile.)
>

Is this fix already on U8 ? I don't seem to get out of mkinitrd (which is
triggered by kernel make install).

OTOH, still on my laptop (P4/UP) I'm getting this very often:

RTNL: assertion failed at net/ipv4/devinet.c (1049)
 [<c0104ee4>] dump_stack+0x1e/0x20 (20)
 [<c02afa2b>] inet_dump_ifaddr+0x135/0x13a (52)
 [<c027a533>] rtnetlink_dump_all+0x92/0xaa (40)
 [<c028117f>] netlink_dump+0x6c/0x211 (56)
 [<c0280f97>] netlink_recvmsg+0x209/0x21b (92)
 [<c0268a40>] sock_recvmsg+0xcc/0xf0 (248)
 [<c026a4cc>] sys_recvmsg+0x110/0x1fb (284)
 [<c026a628>] sys_socketcall+0x71/0x234 (68)
 [<c01040a9>] sysenter_past_esp+0x52/0x71 (-8124)
preempt count: 00000001
. 1-level deep critical section nesting:
.. entry 1: print_traces+0x16/0x4a / (dump_stack+0x1e/0x20)


And I also found this once:

------------[ cut here ]------------
kernel BUG at lib/rwsem-generic.c:598!
invalid operand: 0000 [#1]
PREEMPT
Modules linked in: realtime commoncap snd_seq_oss snd_seq_midi_event
snd_seq snd_pcm_oss snd_mixer_oss snd_usb_usx2y snd_usb_lib snd_rawmidi
snd_seq_device snd_hwdep snd_ali5451 snd_ac97_codec snd_pcm snd_timer
snd_page_alloc snd soundcore prism2_cs p80211 ds yenta_socket pcmcia_core
natsemi crc32 loop subfs evdev ohci_hcd usbcore thermal processor fan
button battery ac
CPU:    0
EIP:    0060:[<c01b7e30>]    Not tainted VLI
EFLAGS: 00010202   (2.6.9-rc4-mm1-RT-U8.0)
EIP is at up_write+0x1d4/0x202
eax: d4edc000   ebx: e003f967   ecx: d4eb8d40   edx: dee04020
esi: de9b4214   edi: de9b443c   ebp: d4eddfcc   esp: d4eddfac
ds: 007b   es: 007b   ss: 0068   preempt: 00000001
Process loop0 (pid: 6672, threadinfo=d4edc000 task=d4eb8d40)
Stack: c0113b01 00000001 c0384d90 c0384d60 00000282 e003f967 de9b4000
de9b443c
       d4eddfec e003f9c8 d4eb8d40 ffffffec 00000000 e003f967 00000000
00000000
       00000000 c0102305 de9b4000 00000000 00000000
Call Trace:
 [<c0104eb0>] show_stack+0x80/0x96 (28)
 [<c010504b>] show_registers+0x165/0x1de (56)
 [<c010525d>] die+0xf6/0x191 (64)
 [<c0105797>] do_invalid_op+0x10b/0x10d (188)
 [<c0104b0d>] error_code+0x2d/0x38 (100)
 [<e003f9c8>] loop_thread+0x61/0x11b [loop] (32)
 [<c0102305>] kernel_thread_helper+0x5/0xb (722608148)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: die+0x3a/0x191 / (do_invalid_op+0x10b/0x10d)
.. entry 2: print_traces+0x16/0x4a / (show_stack+0x80/0x96)

Code: e8 af f9 ff ff 89 f8 e8 f1 af f5 ff e9 35 ff ff ff 0f 0b a5 00 43 e4
2c c0 e9 da fe ff ff 0f 0b a4 00 43 e4 2c c0 e9 c4 fe ff ff <0f> 0b 56 02
cf 70 2d c0 e9 3c fe ff ff e8 d7 56 10 00 e9 22 ff


(config.gz is attached)

Bye now.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

[-- Attachment #2: config-2.6.9-rc4-mm1-RT-U8.0.gz --]
[-- Type: application/x-gzip-compressed, Size: 8010 bytes --]

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

* Rui Nuno Capela <rncbc@rncbc.org> wrote:

> Ingo Molnar wrote:
> >
> >> Changes since -U7:
> >>
> >
> > - fix block-loopback assert reported by Mark H Johnson, Matthew L
> >   Foster and Rui Nuno Capela. (usually triggers during 'make install'
> >   of a kernel compile.)
> >
> 
> Is this fix already on U8 ? I don't seem to get out of mkinitrd (which
> is triggered by kernel make install).

please re-download -U8, i've updated it a couple of minutes after
uploading it, but apparently not fast enough :-| Sorry!

> OTOH, still on my laptop (P4/UP) I'm getting this very often:
> 
> RTNL: assertion failed at net/ipv4/devinet.c (1049)

yeah - this too was an oversight i fixed in the latest upload.

> ------------[ cut here ]------------
> kernel BUG at lib/rwsem-generic.c:598!

>  [<c0104b0d>] error_code+0x2d/0x38 (100)
>  [<e003f9c8>] loop_thread+0x61/0x11b [loop] (32)
>  [<c0102305>] kernel_thread_helper+0x5/0xb (722608148)

yes, this is the loopback fix. Please-retry with the latest patch.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Rui Nuno Capela]

Ingo Molnar wrote:
>
>Rui Nuno Capela wrote:
>> >
>> > - fix block-loopback assert reported by Mark H Johnson, Matthew L
>> >   Foster and Rui Nuno Capela. (usually triggers during 'make install'
>> >   of a kernel compile.)
>> >
>>
>> Is this fix already on U8 ? I don't seem to get out of mkinitrd (which
>> is triggered by kernel make install).
>
> please re-download -U8, i've updated it a couple of minutes after
> uploading it, but apparently not fast enough :-| Sorry!
>

OK. No problem.... and yes, mkinitrd (make install) works again.


>> OTOH, still on my laptop (P4/UP) I'm getting this very often:
>>
>> RTNL: assertion failed at net/ipv4/devinet.c (1049)
>
> yeah - this too was an oversight i fixed in the latest upload.

I don't think so. I still see plenty of those here.

Is there an even more recent U8? I think you should consider add some dot
numbering to each of the uploads... ;)

Bye now.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

* Rui Nuno Capela <rncbc@rncbc.org> wrote:

> > please re-download -U8, i've updated it a couple of minutes after
> > uploading it, but apparently not fast enough :-| Sorry!
> >
> 
> OK. No problem.... and yes, mkinitrd (make install) works again.

good.

> >> RTNL: assertion failed at net/ipv4/devinet.c (1049)
> >
> > yeah - this too was an oversight i fixed in the latest upload.
> 
> I don't think so. I still see plenty of those here.
> 
> Is there an even more recent U8? I think you should consider add some
> dot numbering to each of the uploads... ;)

indeed this most likely means there's a newer update :-| Please 
double-check that the one you have is:

 $ md5sum realtime-preempt-2.6.9-rc4-mm1-U8
 b59ae00ca0f45f545519348113af5c4f  realtime-preempt-2.6.9-rc4-mm1-U8

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Rui Nuno Capela]

Ingo Molnar wrote:
>
>> >> RTNL: assertion failed at net/ipv4/devinet.c (1049)
>> >
>> > yeah - this too was an oversight i fixed in the latest upload.
>>
>> I don't think so. I still see plenty of those here.
>>
>> Is there an even more recent U8? I think you should consider add some
>> dot numbering to each of the uploads... ;)
>
> indeed this most likely means there's a newer update :-| Please
> double-check that the one you have is:
>
>  $ md5sum realtime-preempt-2.6.9-rc4-mm1-U8
>  b59ae00ca0f45f545519348113af5c4f  realtime-preempt-2.6.9-rc4-mm1-U8
>

That was it. Thanks.

Now's some bad news:

I getting the dump below, this time while plugging a flash memory stick,
but right after that the system starts to behave preety bad and
increasingly unresponsive. An hard-boot is almost the end of the (short)
story :(

(e.g. running jackd also hoses the complete system in no reproducible
amount of time--sometimes short, other times long, like a random
time-bomb).


ohci_hcd 0000:00:0f.0: wakeup
usb 2-1: new full speed USB device using address 2
Initializing USB Mass Storage driver...
scsi0 : SCSI emulation for USB Mass Storage devices
usbcore: registered new driver usb-storage
USB Mass Storage support registered.
usb-storage: device found at 2
usb-storage: waiting for device to settle before scanning
------------[ cut here ]------------
kernel BUG at lib/rwsem-generic.c:598!
invalid operand: 0000 [#1]
PREEMPT
Modules linked in: usb_storage vfat fat udf isofs nls_base realtime
commoncap snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss
snd_usb_usx2y snd_usb_lib snd_rawmidi snd_seq_device snd_hwdep snd_ali5451
snd_ac97_codec snd_pcm snd_timer snd_page_alloc snd soundcore prism2_cs
p80211 ds yenta_socket pcmcia_core natsemi crc32 loop subfs evdev ohci_hcd
usbcore thermal processor fan button battery ac
CPU:    0
EIP:    0060:[<c01b7e30>]    Not tainted VLI
EFLAGS: 00010206   (2.6.9-rc4-mm1-RT-U8.3)
EIP is at up_write+0x1d4/0x202
eax: d2b2a000   ebx: 00000292   ecx: d2afe980   edx: d2ad4f40
esi: d7b83b24   edi: dcb21000   ebp: d2b2bd6c   esp: d2b2bd4c
ds: 007b   es: 007b   ss: 0068   preempt: 00000001
Process usb-stor (pid: 6699, threadinfo=d2b2a000 task=d2ad48b0)
Stack: d2ad48b0 d2b2bd78 c02bea7f 00000001 d2ad48b0 00000292 d2afe980
dcb21000
       d2b2bd84 e01ca139 d2afe980 d2b2bd84 00000292 dcb21138 d2b2bdac
c022ed18
       d2afe980 c022ef1c c0231679 00000000 d2afe9d4 d2afe980 d2aa3800
dcb21000
Call Trace:
 [<c0104eb0>] show_stack+0x80/0x96 (28)
 [<c010504b>] show_registers+0x165/0x1de (56)
 [<c010525d>] die+0xf6/0x191 (64)
 [<c0105797>] do_invalid_op+0x10b/0x10d (188)
 [<c0104b0d>] error_code+0x2d/0x38 (100)
 [<e01ca139>] queuecommand+0x70/0x7c [usb_storage] (24)
 [<c022ed18>] scsi_dispatch_cmd+0x168/0x218 (40)
 [<c02342ed>] scsi_request_fn+0x1ee/0x42b (52)
 [<c0205612>] blk_insert_request+0xcd/0xfb (44)
 [<c0232f4f>] scsi_insert_special_req+0x3b/0x3f (28)
 [<c0233181>] scsi_wait_req+0x61/0x94 (60)
 [<c023529c>] scsi_probe_lun+0x8e/0x240 (68)
 [<c023588f>] scsi_probe_and_add_lun+0xb0/0x1be (48)
 [<c0236015>] scsi_scan_target+0xa4/0x123 (60)
 [<c0236121>] scsi_scan_channel+0x8d/0xa4 (48)
 [<c02361b1>] scsi_scan_host_selected+0x79/0xd4 (44)
 [<c023623d>] scsi_scan_host+0x31/0x33 (28)
 [<e01cccbd>] usb_stor_scan_thread+0x144/0x155 [usb_storage] (96)
 [<c0102305>] kernel_thread_helper+0x5/0xb (760037396)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: die+0x3a/0x191 / (do_invalid_op+0x10b/0x10d)
.. entry 2: print_traces+0x16/0x4a / (show_stack+0x80/0x96)

Code: e8 af f9 ff ff 89 f8 e8 f1 af f5 ff e9 35 ff ff ff 0f 0b a5 00 e3 e8
2c c0 e9 da fe ff ff 0f 0b a4 00 e3 e8 2c c0 e9 c4 fe ff ff <0f> 0b 56 02
6f 75 2d c0 e9 3c fe ff ff e8 7f 5b 10 00 e9 22 ff

Bye.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Michal Schmidt]

Ingo Molnar wrote:
> i have released the -U8 Real-Time Preemption patch:
> 
>   http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U8
> 

I'm getting these BUGs when I use netconsole with Real-Time Preemption 
(but netconsole works):

kjournald starting.  Commit interval 5 seconds
BUG: sleeping function called from invalid context kjournald(775) at 
kernel/mutex.c:25
in_atomic():0 [00000000], irqs_disabled():1
  [<c0105fbe>] dump_stack+0x1e/0x20 (20)
  [<c01194d8>] __might_sleep+0xb8/0xd0 (36)
  [<c0130bc0>] _mutex_lock+0x20/0x40 (20)
  [<c02675f7>] netpoll_send_skb+0x37/0xc0 (28)
  [<c0231081>] write_msg+0x41/0x60 (36)
  [<c011c208>] __call_console_drivers+0x58/0x60 (32)
  [<c011c326>] call_console_drivers+0x96/0x140 (40)
  [<c011c6e1>] release_console_sem+0x71/0x100 (36)
  [<c011c5b6>] vprintk+0x116/0x180 (36)
  [<c011c498>] printk+0x18/0x20 (16)
  [<c01a31af>] kjournald+0x8f/0x250 (140)
  [<c01032d1>] kernel_thread_helper+0x5/0x14 (141484052)
preempt count: 00000001
. 1-level deep critical section nesting:
.. entry 1: print_traces+0x18/0x50 / (dump_stack+0x1e/0x20)

EXT3 FS on hda8, <3>BUG: sleeping function called from invalid context 
mount(786) at kernel/mutex.c:25
in_atomic():0 [00000000], irqs_disabled():1
  [<c0105fbe>] dump_stack+0x1e/0x20 (20)
  [<c01194d8>] __might_sleep+0xb8/0xd0 (36)
  [<c0130bc0>] _mutex_lock+0x20/0x40 (20)
  [<c02675f7>] netpoll_send_skb+0x37/0xc0 (28)
  [<c0231081>] write_msg+0x41/0x60 (36)
  [<c011c208>] __call_console_drivers+0x58/0x60 (32)
  [<c011c302>] call_console_drivers+0x72/0x140 (40)
  [<c011c6e1>] release_console_sem+0x71/0x100 (36)
  [<c011c5b6>] vprintk+0x116/0x180 (36)
  [<c011c498>] printk+0x18/0x20 (16)
  [<c01989f2>] ext3_setup_super+0xd2/0x1c0 (80)
  [<c019a5ed>] ext3_remount+0x12d/0x190 (48)
  [<c015d9b0>] do_remount_sb+0xa0/0xf0 (32)
  [<c0173c6d>] do_remount+0x6d/0xc0 (36)
  [<c01745fb>] do_mount+0x19b/0x1b0 (116)
  [<c01749e7>] sys_mount+0x97/0xe0 (48)
  [<c010518f>] syscall_call+0x7/0xb (-8124)
preempt count: 00000001
. 1-level deep critical section nesting:
.. entry 1: print_traces+0x18/0x50 / (dump_stack+0x1e/0x20)

internal journal


I have hacked the sk98lin driver to support netpoll (I sent the patch to 
netdev), so maybe I did something wrong and these BUGs are my own fault.

Does anybody else use netconsole with Real-Time Preemption?

Michal

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

* Michal Schmidt <xschmi00@stud.feec.vutbr.cz> wrote:

> I'm getting these BUGs when I use netconsole with Real-Time Preemption
> (but netconsole works):

you are getting them because interrupts get disabled somewhere in the
path. Do your changes perhaps introduce a local_irq_save() or
local_irq_disable()?

(in PREEMPT_REALTIME spin_lock_irq*() does not disable interrupts for
mutex-based spinlocks, so the only way to get irqs disabled is
explicitly.)

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Michal Schmidt]

[-- Attachment #1: Type: text/plain, Size: 597 bytes --]

Ingo Molnar wrote:
> * Michal Schmidt <xschmi00@stud.feec.vutbr.cz> wrote:
>>I'm getting these BUGs when I use netconsole with Real-Time Preemption
>>(but netconsole works):
> 
> 
> you are getting them because interrupts get disabled somewhere in the
> path. Do your changes perhaps introduce a local_irq_save() or
> local_irq_disable()?
> 

I'm attaching my sk98lin patch. It uses disable_irq(). It's inspired by 
8139too.

> (in PREEMPT_REALTIME spin_lock_irq*() does not disable interrupts for
> mutex-based spinlocks, so the only way to get irqs disabled is
> explicitly.)
> 
> 	Ingo

Michal

[-- Attachment #2: sk98lin-netpoll2.diff --]
[-- Type: text/plain, Size: 1190 bytes --]

diff -Nurp linux-2.6.9/drivers/net/sk98lin/skge.c linux-2.6.9-mich/drivers/net/sk98lin/skge.c
--- linux-2.6.9/drivers/net/sk98lin/skge.c	2004-10-18 23:53:22.000000000 +0200
+++ linux-2.6.9-mich/drivers/net/sk98lin/skge.c	2004-10-20 01:09:07.566181320 +0200
@@ -1126,6 +1126,21 @@ SK_U32		IntSrc;		/* interrupts source re
 		return SkIsrRetHandled;
 } /* SkGeIsrOnePort */
 
+#ifdef CONFIG_NET_POLL_CONTROLLER
+/**
+ * SkGePollController - polling receive, for netconsole
+ * @dev: network device
+ *
+ * Polling receive - used by netconsole and other diagnostic tools
+ * to allow network i/o with interrupts disabled.
+ */
+static void SkGePollController(struct net_device *dev)
+{
+	disable_irq(dev->irq);
+	SkGeIsr(dev->irq, dev, NULL);
+	enable_irq(dev->irq);
+}
+#endif
 
 /****************************************************************************
  *
@@ -4960,6 +4975,9 @@ static int __devinit skge_probe_one(stru
 	dev->set_mac_address =	&SkGeSetMacAddr;
 	dev->do_ioctl =		&SkGeIoctl;
 	dev->change_mtu =	&SkGeChangeMtu;
+#ifdef CONFIG_NET_POLL_CONTROLLER
+	dev->poll_controller =	&SkGePollController;
+#endif
 	dev->flags &= 		~IFF_RUNNING;
 	SET_NETDEV_DEV(dev, &pdev->dev);
 

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

* Michal Schmidt <xschmi00@stud.feec.vutbr.cz> wrote:

> Ingo Molnar wrote:
> >* Michal Schmidt <xschmi00@stud.feec.vutbr.cz> wrote:
> >>I'm getting these BUGs when I use netconsole with Real-Time Preemption
> >>(but netconsole works):
> >
> >
> >you are getting them because interrupts get disabled somewhere in the
> >path. Do your changes perhaps introduce a local_irq_save() or
> >local_irq_disable()?
> >
> 
> I'm attaching my sk98lin patch. It uses disable_irq(). It's inspired
> by 8139too.

disable_irq() should work fine though. (it doesnt disable local
interrupts, it only disables that particular irq line.) So something
else disabled interrupts - ah, netconsole.c itself. Does the patch below
fix things up for you?

	Ingo

--- linux/drivers/net/netconsole.c.orig
+++ linux/drivers/net/netconsole.c
@@ -73,7 +73,9 @@ static void write_msg(struct console *co
 	if (!np.dev)
 		return;
 
+#ifndef CONFIG_PREEMPT_REALTIME
 	local_irq_save(flags);
+#endif
 
 	for(left = len; left; ) {
 		frag = min(left, MAX_PRINT_CHUNK);
@@ -82,7 +84,9 @@ static void write_msg(struct console *co
 		left -= frag;
 	}
 
+#ifndef CONFIG_PREEMPT_REALTIME
 	local_irq_restore(flags);
+#endif
 }
 
 static struct console netconsole = {

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Michal Schmidt]

[-- Attachment #1: Type: text/plain, Size: 1005 bytes --]

Ingo Molnar wrote:
> disable_irq() should work fine though. (it doesnt disable local
> interrupts, it only disables that particular irq line.) So something
> else disabled interrupts - ah, netconsole.c itself. Does the patch below
> fix things up for you?
> 
> 	Ingo
 > [patch snipped]

That patch was not enough. The BUGs were still showing up the same as 
before.
I tried to debug it myself. I've found an interesting thing in 
kernel/printk.c:release_console_sem(). There is the following sequence:

   spin_lock_irqsave(&logbuf_lock, flags);
   /* ... some code ... */
   spin_unlock(&logbuf_lock);
   call_console_drivers(...);
   local_irq_restore(flags);

I know very little about locking, but I didn't like this two-phased 
unlock. So I replaced it with a single spin_unlock_irqrestore. Patch 
attached.
I'm almost certain that there is a reason for the two-phased unlocking 
and that this patch will break something, but so far it works for me. 
netconsole now works without complaining.

Michal

[-- Attachment #2: printk-console-sem.diff --]
[-- Type: text/x-patch, Size: 609 bytes --]

diff -Nurp linux-2.6.9-rc4-mm1-RT-U8/kernel/printk.c linux-2.6.9-rc4-mm1-RT-U8-m/kernel/printk.c
--- linux-2.6.9-rc4-mm1-RT-U8/kernel/printk.c	2004-10-20 22:20:55.000000000 +0200
+++ linux-2.6.9-rc4-mm1-RT-U8-m/kernel/printk.c	2004-10-20 22:24:40.000000000 +0200
@@ -646,9 +646,8 @@ void release_console_sem(void)
 		_con_start = con_start;
 		_log_end = log_end;
 		con_start = log_end;		/* Flush */
-		spin_unlock(&logbuf_lock);
+		spin_unlock_irqrestore(&logbuf_lock, flags);
 		call_console_drivers(_con_start, _log_end);
-		local_irq_restore(flags);
 	}
 	console_locked = 0;
 	console_may_schedule = 0;

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

* Michal Schmidt <xschmi00@stud.feec.vutbr.cz> wrote:

> That patch was not enough. The BUGs were still showing up the same as
> before. I tried to debug it myself. I've found an interesting thing in
> kernel/printk.c:release_console_sem(). There is the following
> sequence:
> 
>   spin_lock_irqsave(&logbuf_lock, flags);
>   /* ... some code ... */
>   spin_unlock(&logbuf_lock);
>   call_console_drivers(...);
>   local_irq_restore(flags);
> 
> I know very little about locking, but I didn't like this two-phased
> unlock. So I replaced it with a single spin_unlock_irqrestore. Patch
> attached. I'm almost certain that there is a reason for the two-phased
> unlocking and that this patch will break something, but so far it
> works for me.  netconsole now works without complaining.

ah, indeed. Note that this is still not enough - please try to add a
local_irq_enable() to netconsole.c's console-write function - does that
fix it equally well for you?

the reason is that if we crash within an irqs-off section then
netconsole will still be called with interrupts disabled and will
trigger the assert.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> ah, indeed. Note that this is still not enough - please try to add a
> local_irq_enable() to netconsole.c's console-write function - does
> that fix it equally well for you?
> 
> the reason is that if we crash within an irqs-off section then
> netconsole will still be called with interrupts disabled and will
> trigger the assert.

i've added your patch to my tree, plus the extra local_irq_enable(),
this should also fix fbcon - so no changes needed to netconsole.c. All
of these problems will go away if/when the console code goes away from
raw spinlocks.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> 
> * Ingo Molnar <mingo@elte.hu> wrote:
> 
> > ah, indeed. Note that this is still not enough - please try to add a
> > local_irq_enable() to netconsole.c's console-write function - does
> > that fix it equally well for you?
> > 
> > the reason is that if we crash within an irqs-off section then
> > netconsole will still be called with interrupts disabled and will
> > trigger the assert.
> 
> i've added your patch to my tree, plus the extra local_irq_enable(),
> this should also fix fbcon - so no changes needed to netconsole.c. All
> of these problems will go away if/when the console code goes away from
> raw spinlocks.

actually ... i think i'll add the local_irq_enable() to netconsole.c and
fbcon, that way the VGA and serial consoles can still keep interrupts
disabled. That could make the difference between a debuggable and
undebuggable crash ...

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Florian Schmidt]

On Wed, 20 Oct 2004 11:45:08 +0200
Ingo Molnar <mingo@elte.hu> wrote:

>   http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U8

Hi,

i just wanted to let you know that with U8 i still experience the "pauses" i
reported on U6, too. I would guess that it's some scheduler thing as jackd
running SCHED_FIFO and all its clients (at least the audio threads running
SCHED_FIFO) are not affected by the pauses (i don't see any xruns from jackd
and audio processing happily goes along without audible dropouts).

Also it seems that /proc/sys/kernel/trace_enabled == 1 is not the only thing
being able to trigger the pauses. With U6 i also experienced them with
trace_enabled == 0. I have to add though that it took quite a while for them
to kick in (hours) after setting trace_enabled to 0. So my conclusion is
that trace_enabled == 1 just increases the probability of such pauses by
several magnitudes (with 1 i get about one of these pauses per 2-10 minutes,
with 0 it took several hours for the first pause to occur and then they
stayed less frequent than with 1).

Ah and i forgot: dmesg -n 1 does not help..

flo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

* Florian Schmidt <mista.tapas@gmx.net> wrote:

> i just wanted to let you know that with U8 i still experience the
> "pauses" i reported on U6, too. I would guess that it's some scheduler
> thing as jackd running SCHED_FIFO and all its clients (at least the
> audio threads running SCHED_FIFO) are not affected by the pauses (i
> don't see any xruns from jackd and audio processing happily goes along
> without audible dropouts).

ok.

> Also it seems that /proc/sys/kernel/trace_enabled == 1 is not the only
> thing being able to trigger the pauses. With U6 i also experienced
> them with trace_enabled == 0. I have to add though that it took quite
> a while for them to kick in (hours) after setting trace_enabled to 0.
> So my conclusion is that trace_enabled == 1 just increases the
> probability of such pauses by several magnitudes (with 1 i get about
> one of these pauses per 2-10 minutes, with 0 it took several hours for
> the first pause to occur and then they stayed less frequent than with
> 1).

i dont think it's caused by trace_enabled - the trace you sent last time
clearly showed erratic behavior. There's one piece of code i suspect in
particular - could you try the patch below ontop of -U8? (i have
compile- and boot- tested it)

	Ingo

--- linux/kernel/sched.c.orig
+++ linux/kernel/sched.c
@@ -2764,6 +2764,8 @@ need_resched:
 		else
 			deactivate_task(prev, rq);
 	}
+	if (preempt_count() & PREEMPT_ACTIVE)
+		sub_preempt_count(PREEMPT_ACTIVE);
 	if (unlikely(prev->flags & PF_DEAD)) {
 		BUG_ON(prev->state != TASK_RUNNING);
 		prev->state = __TASK_DEAD;
@@ -2940,6 +2942,7 @@ asmlinkage void __sched preempt_schedule
 		return;
 
 need_resched:
+	local_irq_disable();
 	add_preempt_count(PREEMPT_ACTIVE);
 	/*
 	 * We keep the big kernel semaphore locked, but we
@@ -2950,11 +2953,10 @@ need_resched:
 	saved_lock_depth = task->lock_depth;
 	task->lock_depth = -1;
 #endif
-	schedule();
+	__schedule();
 #ifdef CONFIG_PREEMPT_BKL
 	task->lock_depth = saved_lock_depth;
 #endif
-	sub_preempt_count(PREEMPT_ACTIVE);
 
 	/* we could miss a preemption opportunity between schedule and now */
 	barrier();
@@ -3002,7 +3004,6 @@ need_resched:
 #ifdef CONFIG_PREEMPT_BKL
 	task->lock_depth = saved_lock_depth;
 #endif
-	sub_preempt_count(PREEMPT_ACTIVE);
 
 	/* we could miss a preemption opportunity between schedule and now */
 	barrier();
@@ -3842,9 +3843,9 @@ static inline void __cond_resched(void)
 	if (preempt_count() & PREEMPT_ACTIVE)
 		return;
 	do {
+		local_irq_disable();
 		add_preempt_count(PREEMPT_ACTIVE);
-		schedule();
-		sub_preempt_count(PREEMPT_ACTIVE);
+		__schedule();
 	} while (need_resched());
 }
 

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Florian Schmidt]

On Wed, 20 Oct 2004 14:55:00 +0200
Ingo Molnar <mingo@elte.hu> wrote:

> i dont think it's caused by trace_enabled - the trace you sent last time
> clearly showed erratic behavior. There's one piece of code i suspect in
> particular - could you try the patch below ontop of -U8? (i have
> compile- and boot- tested it)

mango:/usr/src/linux-2.6.9-rc4-mm1-U8# patch -p1 </home/tapas/foo.patch 
patching file kernel/sched.c
Hunk #5 succeeded at 3843 with fuzz 1.

building anyways, reporting later..

flo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

* Florian Schmidt <mista.tapas@gmx.net> wrote:

> > i dont think it's caused by trace_enabled - the trace you sent last time
> > clearly showed erratic behavior. There's one piece of code i suspect in
> > particular - could you try the patch below ontop of -U8? (i have
> > compile- and boot- tested it)
> 
> mango:/usr/src/linux-2.6.9-rc4-mm1-U8# patch -p1 </home/tapas/foo.patch 
> patching file kernel/sched.c
> Hunk #5 succeeded at 3843 with fuzz 1.
> 
> building anyways, reporting later..

never worry about a fuzz when applying patches - as long as you dont get
a reject it should be ok.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Florian Schmidt]

On Wed, 20 Oct 2004 15:25:07 +0200
Florian Schmidt <mista.tapas@gmx.net> wrote:

> On Wed, 20 Oct 2004 14:55:00 +0200
> Ingo Molnar <mingo@elte.hu> wrote:
> 
> > i dont think it's caused by trace_enabled - the trace you sent last time
> > clearly showed erratic behavior. There's one piece of code i suspect in
> > particular - could you try the patch below ontop of -U8? (i have
> > compile- and boot- tested it)
> 
> mango:/usr/src/linux-2.6.9-rc4-mm1-U8# patch -p1 </home/tapas/foo.patch 
> patching file kernel/sched.c
> Hunk #5 succeeded at 3843 with fuzz 1.
> 
> building anyways, reporting later..

Hi,

it seems that the pauses went away with that patch. The system is showing a
different weird behaviour now. On last bootup the machine slowly died away
(first my email program froze upon checking for mail, then starting top
would just hang the respective xterm. ps still ran and procuced output [i
didn't capture it though, doh], other stuff would hang, too. upon
ctrl-alt-bkspc to kill the x server, it all locked up.. i have no serial
console or other machine to test if it was still up in any way.

And on this bootup the pauses are still gone, but as soon as i echo'ed 1
into trace_enabled the mouse started to become very skippy (update freq at
about 3hz). Keyboard is fine though.. putting trace_enabled back to 0
doesn't fix it. I suppose it's just a matter of time until the next lockup.
We'll see though..

Syslog only sees critical section timing reports, no BUG's afaics.

flo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

* Florian Schmidt <mista.tapas@gmx.net> wrote:

> Hi,
> 
> it seems that the pauses went away with that patch. [...]

great! I've uploaded -U8.1 with this fix included:

  http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U8.1

> And on this bootup the pauses are still gone, but as soon as i echo'ed
> 1 into trace_enabled the mouse started to become very skippy (update
> freq at about 3hz). Keyboard is fine though.. putting trace_enabled
> back to 0 doesn't fix it. I suppose it's just a matter of time until
> the next lockup. We'll see though..
> 
> Syslog only sees critical section timing reports, no BUG's afaics.

note that the keyboard and USB interrupts are SCHED_OTHER by default, so
they could be delayed quite long depending on the workload. To avoid
that i'd suggest to:

        chrt --fifo --pid 30 `pidof 'IRQ 1'`
        chrt --fifo --pid 30 `pidof 'IRQ 12'`

(do this for every IRQ you have for input devices.) This puts them below
jackd's priority (which is FIFO 50 iirc) but above all SCHED_OTHER
tasks. The soundcard IRQ i guess you have chrt-ed already?

or did you have them on SCHED_FIFO already?

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Florian Schmidt]

On Wed, 20 Oct 2004 16:18:22 +0200
Ingo Molnar <mingo@elte.hu> wrote:

> note that the keyboard and USB interrupts are SCHED_OTHER by default, so
> they could be delayed quite long depending on the workload. To avoid
> that i'd suggest to:
> 
>         chrt --fifo --pid 30 `pidof 'IRQ 1'`
>         chrt --fifo --pid 30 `pidof 'IRQ 12'`
> 
> (do this for every IRQ you have for input devices.) This puts them below
> jackd's priority (which is FIFO 50 iirc) but above all SCHED_OTHER
> tasks. The soundcard IRQ i guess you have chrt-ed already?
> 
> or did you have them on SCHED_FIFO already?

setting them to SCHED_FIFO even with a prio of 99 won't help. will try
rebooting to see if it's reproducable 

flo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Florian Schmidt]

On Wed, 20 Oct 2004 16:53:26 +0200
Florian Schmidt <mista.tapas@gmx.net> wrote:

> setting them to SCHED_FIFO even with a prio of 99 won't help. will try
> rebooting to see if it's reproducable 
> 
> flo

ok, it seems it was coincidence that the mouse skipping started at the time
of my echo 1 > trace_enabled.. This time it just started sometime after
boot. The scheduler class of the different irq's seems to have no influence
[i experimented with SCHED_FIFO and SCHED_OTHER for irq 1, 8, 12 and 3 and 10
[the latter two are my soundcards irq's].

~$ cat /proc/interrupts 
           CPU0       
  0:     480317          XT-PIC  timer  0/80317
  1:       1731          XT-PIC  i8042  0/1731
  2:          0          XT-PIC  cascade  0/0
  3:      10828          XT-PIC  CS46XX  0/10828
  5:        390          XT-PIC  eth0  0/390
  8:          4          XT-PIC  rtc  0/4
 10:     251556          XT-PIC  SiS SI7012  0/51556
 12:      16605          XT-PIC  i8042  0/16604
 14:       1151          XT-PIC  ide0  0/1151
 15:      26537          XT-PIC  ide1  0/26537
NMI:          0 
ERR:          0

Btw: i just experienced two pauses again, so the patch didn't really fix it
:( hrmm..

I get the feeling that something indeterministic is going on. Still no BUG's
in either dmesg output nor in the syslog.

 flo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Lee Revell]

On Wed, 2004-10-20 at 10:18, Ingo Molnar wrote:
> note that the keyboard and USB interrupts are SCHED_OTHER by default, so
> they could be delayed quite long depending on the workload.

Why is this the default behavior?  It seems like you would want all IRQ
threads to be SCHED_FIFO by default.   Otherwise it seems like the
scheduler could decide to run a normal userspace process (like, say, X)
while an IRQ thread is runnable.

Is it really a good idea for IRQ threads to be subject to the whims of
the scheduler?

Also, on modern machines, this would effectively make all IRQ threads
SCHED_OTHER because the USB port shares an interrupt with everything:

  0:   36676353          XT-PIC  timer  0/76353
  1:       8759          XT-PIC  i8042  5/8759
  2:          0          XT-PIC  cascade  0/0
  8:          4          XT-PIC  rtc  0/4
 10:          0          XT-PIC  uhci_hcd  0/0
 11:     210713          XT-PIC  uhci_hcd, eth0  0/10713
 12:          0          XT-PIC  uhci_hcd  0/0
 15:      79277          XT-PIC  ide1  0/79276
NMI:          0
ERR:          0

Lee

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Lorenzo Allegrucci]

On Wednesday 20 October 2004 11:45, Ingo Molnar wrote:
> 
> i have released the -U8 Real-Time Preemption patch:
> 
>   http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U8

------------[ cut here ]------------
kernel BUG at lib/rwsem-generic.c:598!
invalid operand: 0000 [#1]
PREEMPT 
Modules linked in: irtty_sir sir_dev irda crc_ccitt usbcore lp ipv6 dm_mod it87 i2c_isa i2c
CPU:    0
EIP:    0060:[<c01dfa5b>]    Not tainted VLI
EFLAGS: 00010287   (2.6.9-rc4-mm1-RT-U8) 
EIP is at up_write+0x1eb/0x200
eax: de848000   ebx: df3a255c   ecx: 0000001f   edx: c14e3260
esi: df3a24c8   edi: de848000   ebp: de849f7c   esp: de849f5c
ds: 007b   es: 007b   ss: 0068   preempt: 00000001
Process kIrDAd (pid: 1295, threadinfo=de848000 task=df3d6110)
Stack: e09119bb de849f74 c0111590 df3a2400 0000001f df3a255c e0915000 de848000 
       de849f94 e09119bb df3a2400 00000282 de849fc4 00000000 de849fec e0911ae2 
       e09129c2 de849fb8 00000000 df3d6110 c01148d0 00000000 00000000 de84ff08 
Call Trace:
 [<e09119bb>] run_irda_queue+0x5b/0xd0 [sir_dev] (4)
 [<c0111590>] mcount+0x14/0x18 (8)
 [<e09119bb>] run_irda_queue+0x5b/0xd0 [sir_dev] (28)
 [<e0911ae2>] irda_thread+0xb2/0xf0 [sir_dev] (24)
 [<c01148d0>] default_wake_function+0x0/0x20 (20)
 [<c010612a>] ret_from_fork+0x6/0x14 (16)
 [<c01148d0>] default_wake_function+0x0/0x20 (16)
 [<e0911a30>] irda_thread+0x0/0xf0 [sir_dev] (24)
 [<c0104319>] kernel_thread_helper+0x5/0xc (12)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: die+0x3f/0x1a0 / (do_invalid_op+0x106/0x110)
.. entry 2: print_traces+0x1d/0x80 / (show_stack+0x83/0xa0)

Code: ff e9 44 ff ff ff 0f 0b a5 00 13 ce 2e c0 eb b6 0f 0b a4 00 13 ce 2e c0 eb a5 c7 04 2
 (events/0/3/CPU#0): new 785 us maximum-latency critical section.
 => started at timestamp 193385865: <kernel_fpu_begin+0x21/0x60>
 =>   ended at timestamp 193386650: <_mmx_memcpy+0x131/0x180>
 [<c012f070>] sub_preempt_count+0x60/0x90 (4)
 [<c012ed5e>] check_preempt_timing+0x15e/0x270 (8)
 [<c01e1a11>] _mmx_memcpy+0x131/0x180 (8)
 [<c012f070>] sub_preempt_count+0x60/0x90 (64)
 [<c01e1a11>] _mmx_memcpy+0x131/0x180 (8)
 [<c01e1a11>] _mmx_memcpy+0x131/0x180 (16)
 [<c01ee1d0>] vgacon_save_screen+0x80/0x90 (28)
 [<c021ba89>] redraw_screen+0x199/0x270 (28)
 [<c012e4ed>] __mcount+0x1d/0x20 (12)
 [<c0216d61>] complete_change_console+0x11/0x100 (4)
 [<c021ec86>] console_callback+0xe6/0xf0 (4)
 [<c0216d89>] complete_change_console+0x39/0x100 (28)
 [<c021ec86>] console_callback+0xe6/0xf0 (28)
 [<c0128e63>] worker_thread+0x1d3/0x2a0 (20)
 [<c021eba0>] console_callback+0x0/0xf0 (28)
 [<c01148d0>] default_wake_function+0x0/0x20 (28)
 [<c01148d0>] default_wake_function+0x0/0x20 (32)
 [<c0111590>] mcount+0x14/0x18 (12)
 [<c012d26a>] kthread+0xaa/0xb0 (28)
 [<c0128c90>] worker_thread+0x0/0x2a0 (20)
 [<c012d1c0>] kthread+0x0/0xb0 (12)
 [<c0104319>] kernel_thread_helper+0x5/0xc (16)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: kernel_fpu_begin+0x21/0x60 / (_mmx_memcpy+0x36/0x180)
.. entry 2: print_traces+0x1d/0x80 / (dump_stack+0x23/0x30)

 =>   dump-end timestamp 193387119

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

* Lorenzo Allegrucci <l_allegrucci@yahoo.it> wrote:

> Process kIrDAd (pid: 1295, threadinfo=de848000 task=df3d6110)

> Call Trace:
>  [<e09119bb>] run_irda_queue+0x5b/0xd0 [sir_dev] (4)
>  [<c0111590>] mcount+0x14/0x18 (8)
>  [<e09119bb>] run_irda_queue+0x5b/0xd0 [sir_dev] (28)
>  [<e0911ae2>] irda_thread+0xb2/0xf0 [sir_dev] (24)

ok - IRDA too needs fixing. Disable CONFIG_IRDA for the time being.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Adam Heath]

On Wed, 20 Oct 2004, Ingo Molnar wrote:

>
> i have released the -U8 Real-Time Preemption patch:
>
>   http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U8

Grr.  I got this -U8 mail *before* I got the -U7 mail.

All thruout these U# kernels, I get stalls in X(everything pauses, not sure if
remote pings stop).  Nothing shows in dmesg when this occurs.

I had been running rc kernels for awhile, without these problems.  I had not
run any mm kernels, but I have run all the U kernels(but not U7).

Compiling/rebooting.

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Adam Heath]

On Wed, 20 Oct 2004, Adam Heath wrote:

> On Wed, 20 Oct 2004, Ingo Molnar wrote:
>
> >
> > i have released the -U8 Real-Time Preemption patch:
> >
> >   http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U8
>
> Grr.  I got this -U8 mail *before* I got the -U7 mail.
>
> All thruout these U# kernels, I get stalls in X(everything pauses, not sure if
> remote pings stop).  Nothing shows in dmesg when this occurs.

Got some input on this.

Heavy disk and/or network i/o seems to cause the pauses.  Doing a copy over
nfs(writing to disk), or using scp gives me 1-2 MB/s.  If I boot plain rc4, I
get full network speed(10-11 MB/s with nfs, 5-8 MB/s with scp).

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Alexander Batyrshin]

[-- Attachment #1: Type: text/plain, Size: 4723 bytes --]



Ingo Molnar wrote:
> i have released the -U8 Real-Time Preemption patch:
> 
>   http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U8
> 

used i386/defconfig

1. at boot
[...skip...]
hda: 78242976 sectors (40060 MB) w/2048KiB Cache, CHS=16383/255/63, 
UDMA(100)
  hda: hda1 hda2 hda3 hda4 < hda5 >
BUG: semaphore recursion deadlock detected!
.. current task khpsbpkt/723 is already holding c04610c0.
00001f23 00001f2f c04e8de9 00000086 00000009 00000000 c011b552 00000020
        00000400 c03b3efa dfdc9f70 dfdc9f50 0000000c dfdc78f0 c011b430 
c03b3efa
        dfdc9f70 c01052a5 c03b3efa c03b3efa 00000000 dfdc78f0 dfdc78f0 
c01fd364
Call Trace:
  [<c012caa4>] __kernel_text_address+0x2e/0x37 (24)
  [<c01051c9>] show_trace+0x4e/0xcc (12)
  [<c01052c9>] show_stack+0x82/0x97 (36)
  [<c01fd364>] __rwsem_deadlock+0xd9/0x135 (24)
  [<c039e2a0>] down_write_interruptible+0xe6/0x202 (48)
  [<c029dd80>] hpsbpkt_thread+0x2b/0x86 (48)
  [<c029dd55>] hpsbpkt_thread+0x0/0x86 (12)
  [<c01024d1>] kernel_thread_helper+0x5/0xb (4)
preempt count: 00000003
. 3-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: _spin_lock+0x19/0x6d / (0x0)
.. entry 3: print_traces+0x17/0x50 / (0x0)
[...skip...]

2.
if execute
``for i in `seq 1 9999`; do nohup bash >/dev/null 2>&1 & done'',
then you'll get something like:
[...skip...]
Warning: dev (pts0) tty->count(16) != #fd's(8) in tty_open
Warning: dev (pts0) tty->count(16) != #fd's(11) in tty_open
Warning: dev (pts0) tty->count(17) != #fd's(13) in tty_open
Warning: dev (pts0) tty->count(18) != #fd's(16) in tty_open
Warning: dev (pts0) tty->count(18) != #fd's(16) in tty_open
Warning: dev (pts0) tty->count(18) != #fd's(16) in tty_open
Warning: dev (pts0) tty->count(19) != #fd's(16) in tty_open
Warning: dev (pts0) tty->count(19) != #fd's(16) in release_dev
Warning: dev (pts0) tty->count(18) != #fd's(16) in tty_open
Warning: dev (pts0) tty->count(18) != #fd's(16) in tty_open
Warning: dev (pts0) tty->count(19) != #fd's(17) in tty_open
Warning: dev (pts0) tty->count(19) != #fd's(17) in release_dev
Warning: dev (pts0) tty->count(18) != #fd's(17) in tty_open
Warning: dev (pts0) tty->count(18) != #fd's(17) in tty_open
Warning: dev (pts0) tty->count(19) != #fd's(17) in release_dev
Warning: dev (pts0) tty->count(17) != #fd's(18) in release_dev
Warning: dev (pts0) tty->count(16) != #fd's(19) in release_dev
Warning: dev (pts0) tty->count(15) != #fd's(18) in release_dev
Warning: dev (pts0) tty->count(14) != #fd's(18) in release_dev
Warning: dev (pts0) tty->count(14) != #fd's(18) in release_dev
Warning: dev (pts0) tty->count(13) != #fd's(18) in tty_open
Warning: dev (pts0) tty->count(14) != #fd's(19) in release_dev
Warning: dev (pts0) tty->count(13) != #fd's(18) in release_dev
Warning: dev (pts0) tty->count(13) != #fd's(17) in release_dev
Warning: dev (pts0) tty->count(12) != #fd's(18) in tty_open
Warning: dev (pts0) tty->count(12) != #fd's(18) in release_dev
Warning: dev (pts0) tty->count(12) != #fd's(18) in release_dev
Warning: dev (pts0) tty->count(28) != #fd's(16) in tty_open
Warning: dev (pts0) tty->count(528) != #fd's(13) in tty_open
Warning: dev (pts0) tty->count(528) != #fd's(13) in release_dev
Warning: dev (pts0) tty->count(527) != #fd's(12) in tty_open
Warning: dev (pts0) tty->count(528) != #fd's(12) in release_dev
Warning: dev (pts0) tty->count(538) != #fd's(28) in release_dev
Warning: dev (pts0) tty->count(537) != #fd's(528) in tty_open
Warning: dev (pts0) tty->count(537) != #fd's(527) in release_dev
Warning: dev (pts0) tty->count(536) != #fd's(527) in tty_open
Warning: dev (pts0) tty->count(536) != #fd's(527) in tty_open
Warning: dev (pts0) tty->count(536) != #fd's(527) in release_dev
Warning: dev (pts0) tty->count(535) != #fd's(527) in tty_open
Warning: dev (pts0) tty->count(535) != #fd's(530) in tty_open
[...skip...]
Warning: dev (pts0) tty->count(11) != #fd's(71) in release_dev
Warning: dev (pts0) tty->count(10) != #fd's(71) in release_dev
Warning: dev (pts0) tty->count(9) != #fd's(71) in release_dev
Warning: dev (pts0) tty->count(8) != #fd's(71) in release_dev
Warning: dev (pts0) tty->count(7) != #fd's(71) in release_dev
Warning: dev (pts0) tty->count(6) != #fd's(71) in release_dev
Warning: dev (pts0) tty->count(5) != #fd's(71) in release_dev
Warning: dev (pts0) tty->count(4) != #fd's(71) in release_dev
Warning: dev (pts0) tty->count(3) != #fd's(71) in release_dev
eggcups: page allocation failure. order:1, mode:0x20
  [<c01439d8>] __alloc_pages+0x228/0x3ff (8)
  [<c0143bce>] __get_free_pages+0x1f/0x3b (68)
  [<c014719c>] kmem_getpages+0x25/0xe0 (8)
[...skip...]

I'v tested it against linux-2.6.9-rc4-mm1 => all was ok

See full logs in attachments

  -bash

[-- Attachment #2: U8-smp1-0.txt --]
[-- Type: text/plain, Size: 18539 bytes --]

Linux version 2.6.9-rc4-mm1-RT-U8 (bash@devnull.rtsoft) (gcc version 3.2.2 20030222 (Red Hat Linux 3.2.2-5)) #3 SMP Wed Oct 20 16:26:38 MSD 2004
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
 BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000e8000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000001ff30000 (usable)
 BIOS-e820: 000000001ff30000 - 000000001ff40000 (ACPI data)
 BIOS-e820: 000000001ff40000 - 000000001fff0000 (ACPI NVS)
 BIOS-e820: 000000001fff0000 - 0000000020000000 (reserved)
 BIOS-e820: 00000000ffb80000 - 0000000100000000 (reserved)
511MB LOWMEM available.
found SMP MP-table at 000ff780
DMI 2.3 present.
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
Processor #0 15:3 APIC version 20
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
Processor #1 15:3 APIC version 20
Using ACPI for processor (LAPIC) configuration information
Intel MultiProcessor Specification v1.4
    Virtual Wire compatibility mode.
OEM ID: ASUSTeK  Product ID: P4P800SE     APIC at: 0xFEE00000
I/O APIC #13 Version 32 at 0xFEC00000.
Enabling APIC mode:  Flat.  Using 1 I/O APICs
Processors: 2
Built 1 zonelists
Initializing CPU#0
Kernel command line: root=/dev/hda2 console=ttyS0,57600
PID hash table entries: 2048 (order: 11, 32768 bytes)
Detected 2798.899 MHz processor.
Using tsc for high-res timesource
Console: colour VGA+ 80x25
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 512196k/523456k available (2687k kernel code, 10700k reserved, 1089k data, 184k init, 0k highmem)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
monitor/mwait feature present.
using mwait in idle threads.
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 1024K
CPU: Physical Processor ID: 0
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
CPU0: Intel P4/Xeon Extended MCE MSRs (12) available
CPU0: Thermal monitoring enabled
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... OK.
CPU0: Intel(R) Pentium(R) 4 CPU 2.80GHz stepping 03
per-CPU timeslice cutoff: 2926.33 usecs.
task migration cache decay timeout: 3 msecs.
Booting processor 1/1 eip 2000
Initializing CPU#1
monitor/mwait feature present.
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 1024K
CPU: Physical Processor ID: 0
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#1.
CPU1: Intel P4/Xeon Extended MCE MSRs (12) available
CPU1: Thermal monitoring enabled
CPU1: Intel(R) Pentium(R) 4 CPU 2.80GHz stepping 03
Total of 2 processors activated (11108.35 BogoMIPS).
ENABLING IO-APIC IRQs
..TIMER: vector=0x31 pin1=2 pin2=0
checking TSC synchronization across 2 CPUs: passed.
ksoftirqd started up.
Brought up 2 CPUs
ksoftirqd started up.
NET: Registered protocol family 16
PCI: PCI BIOS revision 2.10 entry at 0xf0031, last bus=2
PCI: Using configuration type 1
mtrr: v2.0 (20020519)
Linux Plug and Play Support v0.97 (c) Adam Belay
SCSI subsystem initialized
usbcore: registered new driver usbfs
usbcore: registered new driver hub
PCI: Probing PCI hardware
PCI: Probing PCI hardware (bus 00)
PCI: Ignoring BAR0-3 of IDE controller 0000:00:1f.1
PCI: Transparent bridge - 0000:00:1e.0
PCI: Using IRQ router PIIX/ICH [8086/24d0] at 0000:00:1f.0
PCI->APIC IRQ transform: (B0,I29,P0) -> 16
PCI->APIC IRQ transform: (B0,I29,P1) -> 19
PCI->APIC IRQ transform: (B0,I29,P2) -> 18
PCI->APIC IRQ transform: (B0,I29,P0) -> 16
PCI->APIC IRQ transform: (B0,I29,P3) -> 23
PCI->APIC IRQ transform: (B0,I31,P0) -> 18
PCI->APIC IRQ transform: (B0,I31,P1) -> 17
PCI->APIC IRQ transform: (B0,I31,P1) -> 17
PCI->APIC IRQ transform: (B1,I0,P0) -> 16
PCI->APIC IRQ transform: (B2,I5,P0) -> 22
Machine check exception polling timer started.
audit: initializing netlink socket (disabled)
audit(1098277485.061:0): initialized
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
lp: driver loaded but no devices found
Linux agpgart interface v0.100 (c) Dave Jones
agpgart: Detected an Intel 865 Chipset.
agpgart: Maximum main memory to use for agp memory: 439M
agpgart: AGP aperture is 64M @ 0xf8000000
serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
Serial: 8250/16550 driver $Revision: 1.90 $ 8 ports, IRQ sharing disabled
ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
parport0: PC-style at 0x378 [PCSPP(,...)]
lp0: using parport0 (polling).
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
elevator: using anticipatory as default io scheduler
Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077
eth0: Yukon Gigabit Ethernet 10/100/1000Base-T Adapter
      PrefPort:A  RlmtMode:Check Link State
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
ICH5: IDE controller at PCI slot 0000:00:1f.1
PCI: Enabling device 0000:00:1f.1 (0005 -> 0007)
ICH5: chipset revision 2
ICH5: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0xfc00-0xfc07, BIOS settings: hda:DMA, hdb:pio
    ide1: BM-DMA at 0xfc08-0xfc0f, BIOS settings: hdc:pio, hdd:pio
hda: SAMSUNG SP0411N, ATA DISK drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
hda: max request size: 1024KiB
hda: 78242976 sectors (40060 MB) w/2048KiB Cache, CHS=16383/255/63, UDMA(100)
 hda: hda1 hda2 hda3 hda4 < hda5 >
BUG: semaphore recursion deadlock detected!
.. current task khpsbpkt/723 is already holding c04610c0.
00001f23 00001f2f c04e8de9 00000086 00000009 00000000 c011b552 00000020 
       00000400 c03b3efa dfdc9f70 dfdc9f50 0000000c dfdc78f0 c011b430 c03b3efa 
       dfdc9f70 c01052a5 c03b3efa c03b3efa 00000000 dfdc78f0 dfdc78f0 c01fd364 
Call Trace:
 [<c012caa4>] __kernel_text_address+0x2e/0x37 (24)
 [<c01051c9>] show_trace+0x4e/0xcc (12)
 [<c01052c9>] show_stack+0x82/0x97 (36)
 [<c01fd364>] __rwsem_deadlock+0xd9/0x135 (24)
 [<c039e2a0>] down_write_interruptible+0xe6/0x202 (48)
 [<c029dd80>] hpsbpkt_thread+0x2b/0x86 (48)
 [<c029dd55>] hpsbpkt_thread+0x0/0x86 (12)
 [<c01024d1>] kernel_thread_helper+0x5/0xb (4)
preempt count: 00000003
. 3-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: _spin_lock+0x19/0x6d / (0x0)
.. entry 3: print_traces+0x17/0x50 / (0x0)

------------[ cut here ]------------
kernel BUG at lib/rwsem-generic.c:498!
invalid operand: 0000 [#1]
PREEMPT SMP 
Modules linked in:
CPU:    1
EIP:    0060:[<c039e2b7>]    Not tainted VLI
EFLAGS: 00010002   (2.6.9-rc4-mm1-RT-U8) 
EIP is at down_write_interruptible+0xfd/0x202
eax: 00000001   ebx: c04610c0   ecx: dfdc8000   edx: 00000001
esi: dfdc78f0   edi: c04610c4   ebp: dfdc9fc0   esp: dfdc9fb4
ds: 007b   es: 007b   ss: 0068   preempt: 00000003
Process khpsbpkt (pid: 723, threadinfo=dfdc8000 task=dfdc78f0)
Stack: c04610c0 000001d6 c04610c0 c04610cc c04610cc dfdc78f0 00000002 dfdc8000 
       00000000 00000000 00000000 c029dd80 c03cd95b ffffffff c029dd55 c01024d1 
       00000000 00000000 00000000 
Call Trace:
 [<c029dd80>] hpsbpkt_thread+0x2b/0x86 (48)
 [<c029dd55>] hpsbpkt_thread+0x0/0x86 (12)
 [<c01024d1>] kernel_thread_helper+0x5/0xb (4)
preempt count: 00000004
. 4-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: _spin_lock+0x19/0x6d / (0x0)
.. entry 3: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 4: print_traces+0x17/0x50 / (0x0)

Code: 44 24 0c 89 68 04 89 2a 89 54 24 10 89 1c 24 e8 eb ef e5 ff 85 c0 74 1b a1 2c e0 41 c0 85 c0 74 12 c7 05 2c e0 41 c0 00 00 00 00 <0f> 0b f2 01 cf 73 3c c0 89 9e 1c 06 00 00 9c 58 c1 e8 09 83 f0 
 <3>BUG: scheduling while atomic: khpsbpkt/0x04000002/723
ieee1394: raw1394: /dev/raw1394 device initialized
ehci_hcd 0000:00:1d.7: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB2 EHCI Controller
PCI: Setting latency timer of device 0000:00:1d.7 to 64
ehci_hcd 0000:00:1d.7: irq 23, pci mem 0xfebffc00
ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 1
PCI: cache line size of 128 is not supported by device 0000:00:1d.7
ehci_hcd 0000:00:1d.7: USB 2.0 enabled, EHCI 1.00, driver 2004-May-10
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 8 ports detected
USB Universal Host Controller Interface driver v2.2
uhci_hcd 0000:00:1d.0: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #1
caller is cond_resched+0x5e/0x7f
 [<c039d128>] __sched_text_start+0x6e0/0x730 (8)
 [<c039daa9>] cond_resched+0x5e/0x7f (8)
 [<c0105aff>] do_invalid_op+0x0/0xf7 (56)
 [<c0105aff>] do_invalid_op+0x0/0xf7<7>PCI: Setting latency timer of device 0000:00:1d.0 to 64
uhci_hcd 0000:00:1d.0: irq 16, io base 0xef00
uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
uhci_hcd 0000:00:1d.1: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #2
 (16)
PCI: Setting latency timer of device 0000:00:1d.1 to 64
uhci_hcd 0000:00:1d.1: irq 19, io base 0xef20
uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 3
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 2 ports detected
uhci_hcd 0000:00:1d.2: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #3
 [<c039daa9>] cond_resched+0x5e/0x7f<7>PCI: Setting latency timer of device 0000:00:1d.2 to 64
uhci_hcd 0000:00:1d.2: irq 18, io base 0xef40
uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 4
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 2 ports detected
uhci_hcd 0000:00:1d.3: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #4
 (8)
 [<c039dee7>]<7>PCI: Setting latency timer of device 0000:00:1d.3 to 64
uhci_hcd 0000:00:1d.3: irq 16, io base 0xef80
uhci_hcd 0000:00:1d.3: new USB bus registered, assigned bus number 5
hub 5-0:1.0: USB hub found
hub 5-0:1.0: 2 ports detected
usbcore: registered new driver usblp
drivers/usb/class/usblp.c: v0.13: USB Printer Device Class driver
Initializing USB Mass Storage driver...
usbcore: registered new driver usb-storage
USB Mass Storage support registered.
usbcore: registered new driver usbhid
drivers/usb/input/hid-core.c: v2.0:USB HID core driver
mice: PS/2 mouse device common for all mice
input: AT Translated Set 2 keyboard on isa0060/serio0
 down_read+0x33/0x18a (16)
 [<c0110963>] smp_apic_timer_interrupt+0x6d/0xe2 (12)
 [<c039e92b>] _spin_unlock_irq+0x18/0x2e (8)
 [<c0105aff>] do_invalid_op+0x0/0xf7 (16)
 [<c011bab5>] profile_task_exit+0x15/0x3e (4)
 [<c011dab4>] do_exit+0x18/0x3b1 (20)
 [<c011007b>] flush_tlb_all+0x26/0x65 (12)
 [<c0105aff>] do_invalid_op+0x0/0xf7 (16)
 [<c0105700>] do_divide_error+0x0/0x117 (8)
 [<c039e2b7>] down_write_interruptible+0xfd/0x202 (40)
 [<c01142ca>] fixup_exception+0x16/0x34 (8)
 [<c0105bf4>] do_invalid_op+0xf5/0xf7 (12)
 [<c039e2b7>] down_write_interruptible+0xfd/0x202 (60)
 [<c011b644>] release_console_sem+0x78/0xb7 (28)
 [<c011b552>] vprintk+0x11e/0x15b (28)
 [<c01052c9>] show_stack+0x82/0x97 (56)
 [<c0104f21>] error_code+0x2d/0x38 (12)
 [<c039e2b7>] down_write_interruptible+0xfd/0x202 (52)
 [<c029dd80>] hpsbpkt_thread+0x2b/0x86 (56)
 [<c029dd55>] hpsbpkt_thread+0x0/0x86<6>input: ImPS/2 Generic Wheel Mouse on isa0060/serio1
 (12)
 [<c01024d1>] kernel_thread_helper+0x5/0xb (4)
preempt count: 04000003
Advanced Linux Sound Architecture Driver Version 1.0.6 (Sun Aug 15 07:17:53 2004 UTC).
. 3-level deep critical section nesting:
PCI: Setting latency timer of device 0000:00:1f.5 to 64
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: _spin_lock+0x19/0x6d / (0x0)
.. entry 3: print_traces+0x17/0x50 / (0x0)

note: khpsbpkt[723] exited with preempt_count 2
BUG: scheduling while atomic: khpsbpkt/0x04000002/723
caller is cond_resched+0x5e/0x7f
 [<c039d128>] __sched_text_start+0x6e0/0x730 (8)
 [<c039daa9>] cond_resched+0x5e/0x7f (8)
 [<c01202d4>] __do_softirq+0x36/0x88 (36)
 [<c039daa9>] cond_resched+0x5e/0x7f (44)
 [<c039e068>] down_write+0x2a/0x17c (16)
 [<c011d22e>] exit_notify+0x28/0x896 (40)
 [<c011b552>] vprintk+0x11e/0x15b (24)
 [<c011dd81>] do_exit+0x2e5/0x3b1 (44)
 [<c0105aff>] do_invalid_op+0x0/0xf7 (28)
 [<c0105700>] do_divide_error+0x0/0x117 (8)
 [<c039e2b7>] down_write_interruptible+0xfd/0x202 (40)
 [<c01142ca>] fixup_exception+0x16/0x34 (8)
 [<c0105bf4>] do_invalid_op+0xf5/0xf7 (12)
 [<c039e2b7>] down_write_interruptible+0xfd/0x202 (60)
 [<c011b644>] release_console_sem+0x78/0xb7 (28)
 [<c011b552>] vprintk+0x11e/0x15b (28)
 [<c01052c9>] show_stack+0x82/0x97 (56)
 [<c0104f21>] error_code+0x2d/0x38 (12)
 [<c039e2b7>] down_write_interruptible+0xfd/0x202 (52)
 [<c029dd80>] hpsbpkt_thread+0x2b/0x86 (56)
 [<c029dd55>] hpsbpkt_thread+0x0/0x86 (12)
 [<c01024d1>] kernel_thread_helper+0x5/0xb (4)
preempt count: 04000003
. 3-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: _spin_lock+0x19/0x6d / (0x0)
.. entry 3: print_traces+0x17/0x50 / (0x0)

BUG: scheduling while atomic: khpsbpkt/0x00000002/723
caller is do_exit+0x2ee/0x3b1
 [<c039d128>] __sched_text_start+0x6e0/0x730 (8)
 [<c011dd8a>] do_exit+0x2ee/0x3b1 (8)
 [<c011d486>] exit_notify+0x280/0x896 (12)
 [<c011dd8a>] do_exit+0x2ee/0x3b1 (68)
 [<c0105aff>] do_invalid_op+0x0/0xf7 (28)
 [<c0105700>] do_divide_error+0x0/0x117 (8)
 [<c039e2b7>] down_write_interruptible+0xfd/0x202 (40)
 [<c01142ca>] fixup_exception+0x16/0x34 (8)
 [<c0105bf4>] do_invalid_op+0xf5/0xf7 (12)
 [<c039e2b7>] down_write_interruptible+0xfd/0x202 (60)
 [<c011b644>] release_console_sem+0x78/0xb7<6>intel8x0_measure_ac97_clock: measured 50107 usecs
intel8x0: clocking to 48000
ALSA device list:
  #0: Intel ICH5 with AD1985 at 0xfebff800, irq 17
oprofile: using NMI interrupt.
 (28)
 [<c011b552>] vprintk+0x11e/0x15b<6>NET: Registered protocol family 2
 (28)
 [<c01052c9>] show_stack+0x82/0x97 (56)
 [<c0104f21>] error_code+0x2d/0x38 (12)
 [<c039e2b7>] down_write_interruptible+0xfd/0x202 (52)
 [<c029dd80>] hpsbpkt_thread+0x2b/0x86 (56)
 [<c029dd55>] hpsbpkt_thread+0x0/0x86 (12)
 [<c01024d1>] kernel_thread_helper+0x5/0xb (4)
preempt count: 00000003
. 3-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: _spin_lock+0x19/0x6d / (0x0)
.. entry 3: print_traces+0x17/0x50 / (0x0)

------------[ cut here ]------------
kernel BUG at kernel/sched.c:2768!
invalid operand: 0000 [#2]
PREEMPT SMP 
Modules linked in:
CPU:    1
EIP:    0060:[<c039d094>]    Not tainted VLI
EFLAGS: 00010002   (2.6.9-rc4-mm1-RT-U8) 
EIP is at __sched_text_start+0x64c/0x730
eax: 00000001   ebx: dfdc8000   ecx: dfdc78f0   edx: c14340bc
esi: dfdc78f0   edi: dfdc7a60   ebp: dfdc9e58   esp: dfdc9e08
ds: 007b   es: 007b   ss: 0068   preempt: 00000005
Process khpsbpkt (pid: 723, threadinfo=dfdc8000 task=dfdc78f0)
Stack: dfdc78f0 c1433820 00000002 000002d3 c011d486 c04a6d00 00000011 00000286 
       00000033 dfdc7950 dfdc7994 c1433820 0f5d9fc7 77b2f71c 00000001 dfdc78f0 
       dfdc7a5c c165c780 dfdc78f0 dfdc7db8 dfdc9fc0 c011dd8a dfdc78f0 00000000 
Call Trace:
 [<c011d486>] exit_notify+0x280/0x896 (20)
 [<c011dd8a>] do_exit+0x2ee/0x3b1 (68)
 [<c0105aff>] do_invalid_op+0x0/0xf7 (28)
 [<c0105700>] do_divide_error+0x0/0x117 (8)
 [<c039e2b7>] down_write_interruptible+0xfd/0x202 (40)
 [<c01142ca>] fixup_exception+0x16/0x34 (8)
 [<c0105bf4>] do_invalid_op+0xf5/0xf7 (12)
 [<c039e2b7>] down_write_interruptible+0xfd/0x202 (60)
 [<c011b644>] release_console_sem+0x78/0xb7 (28)
 [<c011b552>] vprintk+0x11e/0x15b (28)
 [<c01052c9>] show_stack+0x82/0x97 (56)
 [<c0104f21>] error_code+0x2d/0x38 (12)
 [<c039e2b7>] down_write_interruptible+0xfd/0x202 (52)
 [<c029dd80>] hpsbpkt_thread+0x2b/0x86 (56)
 [<c029dd55>] hpsbpkt_thread+0x0/0x86 (12)
 [<c01024d1>] kernel_thread_helper+0x5/0xb (4)
preempt count: 00000006
. 6-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: _spin_lock+0x19/0x6d / (0x0)
.. entry 3: __sched_text_start+0x36/0x730 / (0x0)
.. entry 4: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 5: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 6: print_traces+0x17/0x50 / (0x0)

Code: 8b 45 dc 8b 50 08 eb c7 8b 01 85 c0 75 1d 8b 7d ec c7 07 20 00 00 00 89 3c 24 8b 45 dc 89 44 24 04 e8 5b 7b d7 ff e9 fe fa ff ff <0f> 0b d0 0a 82 5c 3b c0 eb d9 8b 7d ec c7 07 00 00 00 00 e9 d9 
 <3>BUG: scheduling while atomic: khpsbpkt/0x04000004/723
caller is cond_resched+0x5e/0x7f
 [<c039d128>] __sched_text_start+0x6e0/0x730 (8)
 [<c039daa9>] cond_resched+0x5e/0x7f (8)
 [<c0116664>] scheduler_tick+0x183/0x517 (28)
 [<c0105aff>] do_invalid_op+0x0/0xf7 (28)
 [<c0105aff>] do_invalid_op+0x0/0xf7 (16)
 [<c039daa9>] cond_resched+0x5e/0x7f (8)
 [<c039dee7>] down_read+0x33/0x18a (16)
 [<c0110963>] smp_apic_timer_interrupt+0x6d/0xe2 (12)
 [<c039e92b>] _spin_unlock_irq+0x18/0x2e (8)
 [<c0105aff>] do_invalid_op+0x0/0xf7 (16)
 [<c011bab5>] profile_task_exit+0x15/0x3e (4)
 [<c011dab4>] do_exit+0x18/0x3b1 (20)
 [<c011007b>] flush_tlb_all+0x26/0x65 (12)
 [<c0105aff>] do_invalid_op+0x0/0xf7 (16)
 [<c0105700>] do_divide_error+0x0/0x117 (8)
 [<c039d094>] __sched_text_start+0x64c/0x730 (40)
 [<c01142ca>] fixup_exception+0x16/0x34 (8)
 [<c0105bf4>] do_invalid_op+0xf5/0xf7 (12)
 [<c039d094>] __sched_text_start+0x64c/0x730 (60)
 [<c011b644>] release_console_sem+0x78/0xb7 (84)
 [<c011b552>] vprintk+0x11e/0x15b (28)
 [<c0104f21>] error_code+0x2d/0x38 (12)
 [<c011007b>] flush_tlb_all+0x26/0x65 (40)
 [<c039d094>] __sched_text_start+0x64c/0x730 (12)
 [<c011d486>] exit_notify+0x280/0x896 (28)
 [<c011dd8a>] do_exit+0x2ee/0x3b1 (68)
 [<c0105aff>] do_invalid_op+0x0/0xf7 (28)
 [<c0105700>] do_divide_error+0x0/0x117 (8)
 [<c039e2b7>] down_write_interruptible+0xfd/0x202 (40)
 [<c01142ca>] fixup_exception+0x16/0x34 (8)
 [<c0105bf4>] do_invalid_op+0xf5/0xf7 (12)
 [<c039e2b7>] down_write_interruptible+0xfd/0x202 (60)
 [<c011b644>] release_console_sem+0x78/0xb7 (28)
 [<c011b552>] vprintk+0x11e/0x15b (28)
 [<c01052c9>] show_stack+0x82/0x97 (56)
 [<c0104f21>] error_code+0x2d/0x38 (12)
 [<c039e2b7>] down_write_interruptible+0xfd/0x202 (52)
 [<c029dd80>] hpsbpkt_thread+0x2b/0x86 (56)
 [<c029dd55>] hpsbpkt_thread+0x0/0x86 (12)
 [<c01024d1>] kernel_thread_helper+0x5/0xb (4)
preempt count: 04000005
. 5-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: _spin_lock+0x19/0x6d / (0x0)
.. entry 3: __sched_text_start+0x36/0x730 / (0x0)
.. entry 4: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 5: print_traces+0x17/0x50 / (0x0)


[-- Attachment #3: U8-smp4-noieee.txt --]
[-- Type: text/plain, Size: 69802 bytes --]

Linux version 2.6.9-rc4-mm1-RT-U8 (bash@devnull.rtsoft) (gcc version 3.2.2 20030222 (Red Hat Linux 3.2.2-5)) #11 SMP Wed Oct 20 18:49:11 MSD 2004
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
 BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000e8000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000001ff30000 (usable)
 BIOS-e820: 000000001ff30000 - 000000001ff40000 (ACPI data)
 BIOS-e820: 000000001ff40000 - 000000001fff0000 (ACPI NVS)
 BIOS-e820: 000000001fff0000 - 0000000020000000 (reserved)
 BIOS-e820: 00000000ffb80000 - 0000000100000000 (reserved)
511MB LOWMEM available.
found SMP MP-table at 000ff780
DMI 2.3 present.
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
Processor #0 15:3 APIC version 20
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
Processor #1 15:3 APIC version 20
ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Enabling APIC mode:  Flat.  Using 1 I/O APICs
Using ACPI (MADT) for SMP configuration information
Built 1 zonelists
Initializing CPU#0
Kernel command line: root=/dev/hda2 console=ttyS0,57600
PID hash table entries: 2048 (order: 11, 32768 bytes)
Detected 2798.919 MHz processor.
Using tsc for high-res timesource
Console: colour VGA+ 80x25
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 512132k/523456k available (2689k kernel code, 10764k reserved, 1111k data, 192k init, 0k highmem)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
monitor/mwait feature present.
using mwait in idle threads.
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 1024K
CPU: Physical Processor ID: 0
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
CPU0: Intel P4/Xeon Extended MCE MSRs (12) available
CPU0: Thermal monitoring enabled
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... OK.
CPU0: Intel(R) Pentium(R) 4 CPU 2.80GHz stepping 03
per-CPU timeslice cutoff: 2926.33 usecs.
task migration cache decay timeout: 3 msecs.
Booting processor 1/1 eip 3000
Initializing CPU#1
monitor/mwait feature present.
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 1024K
CPU: Physical Processor ID: 0
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#1.
CPU1: Intel P4/Xeon Extended MCE MSRs (12) available
CPU1: Thermal monitoring enabled
CPU1: Intel(R) Pentium(R) 4 CPU 2.80GHz stepping 03
Total of 2 processors activated (11108.35 BogoMIPS).
ENABLING IO-APIC IRQs
..TIMER: vector=0x31 pin1=2 pin2=-1
checking TSC synchronization across 2 CPUs: passed.
ksoftirqd started up.
Brought up 2 CPUs
ksoftirqd started up.
NET: Registered protocol family 16
PCI: PCI BIOS revision 2.10 entry at 0xf0031, last bus=2
PCI: Using configuration type 1
mtrr: v2.0 (20020519)
ACPI: Subsystem revision 20040816
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (00:00)
PCI: Probing PCI hardware (bus 00)
PCI: Ignoring BAR0-3 of IDE controller 0000:00:1f.1
PCI: Transparent bridge - 0000:00:1e.0
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 *10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 *10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 6 7 10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 10 *11 12 14 15)
ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 10 *11 12 14 15)
ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 10 *11 12 14 15)
Linux Plug and Play Support v0.97 (c) Adam Belay
SCSI subsystem initialized
usbcore: registered new driver usbfs
usbcore: registered new driver hub
PCI: Using ACPI for IRQ routing
** PCI interrupts are no longer routed automatically.  If this
** causes a device to stop working, it is probably because the
** driver failed to call pci_enable_device().  As a temporary
** workaround, the "pci=routeirq" argument restores the old
** behavior.  If this argument makes the device work again,
** please email the output of "lspci" to bjorn.helgaas@hp.com
** so I can fix the driver.
Machine check exception polling timer started.
audit: initializing netlink socket (disabled)
audit(1098284016.319:0): initialized
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
ACPI: Power Button (FF) [PWRF]
ACPI: Processor [CPU1] (supports C1)
ACPI: Processor [CPU2] (supports C1)
lp: driver loaded but no devices found
Linux agpgart interface v0.100 (c) Dave Jones
agpgart: Detected an Intel 865 Chipset.
agpgart: Maximum main memory to use for agp memory: 439M
agpgart: AGP aperture is 64M @ 0xf8000000
ACPI: PS/2 Keyboard Controller [PS2K] at I/O 0x60, 0x64, irq 1
ACPI: PS/2 Mouse Controller [PS2M] at irq 12
serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
Serial: 8250/16550 driver $Revision: 1.90 $ 8 ports, IRQ sharing disabled
ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
parport0: PC-style at 0x378 [PCSPP(,...)]
lp0: using parport0 (polling).
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
ACPI: Floppy Controller [FDC] at I/O 0x3f0-0x3f5, 0x3f7 irq 6 dma channel 2
elevator: using anticipatory as default io scheduler
Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
ICH5: IDE controller at PCI slot 0000:00:1f.1
PCI: Enabling device 0000:00:1f.1 (0005 -> 0007)
ACPI: PCI interrupt 0000:00:1f.1[A] -> GSI 18 (level, low) -> IRQ 18
ICH5: chipset revision 2
ICH5: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0xfc00-0xfc07, BIOS settings: hda:DMA, hdb:pio
    ide1: BM-DMA at 0xfc08-0xfc0f, BIOS settings: hdc:pio, hdd:pio
hda: SAMSUNG SP0411N, ATA DISK drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
hda: max request size: 1024KiB
hda: 78242976 sectors (40060 MB) w/2048KiB Cache, CHS=16383/255/63, UDMA(100)
 hda: hda1 hda2 hda3 hda4 < hda5 >
ACPI: PCI interrupt 0000:00:1d.7[D] -> GSI 23 (level, low) -> IRQ 23
ehci_hcd 0000:00:1d.7: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB2 EHCI Controller
ehci_hcd 0000:00:1d.7: irq 23, pci mem 0xfebffc00
ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 1
ehci_hcd 0000:00:1d.7: USB 2.0 enabled, EHCI 1.00, driver 2004-May-10
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 8 ports detected
USB Universal Host Controller Interface driver v2.2
ACPI: PCI interrupt 0000:00:1d.0[A] -> GSI 16 (level, low) -> IRQ 16
uhci_hcd 0000:00:1d.0: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #1
uhci_hcd 0000:00:1d.0: irq 16, io base 0xef00
uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
ACPI: PCI interrupt 0000:00:1d.1[B] -> GSI 19 (level, low) -> IRQ 19
uhci_hcd 0000:00:1d.1: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #2
uhci_hcd 0000:00:1d.1: irq 19, io base 0xef20
uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 3
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 2 ports detected
ACPI: PCI interrupt 0000:00:1d.2[C] -> GSI 18 (level, low) -> IRQ 18
uhci_hcd 0000:00:1d.2: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #3
uhci_hcd 0000:00:1d.2: irq 18, io base 0xef40
uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 4
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 2 ports detected
ACPI: PCI interrupt 0000:00:1d.3[A] -> GSI 16 (level, low) -> IRQ 16
uhci_hcd 0000:00:1d.3: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #4
uhci_hcd 0000:00:1d.3: irq 16, io base 0xef80
uhci_hcd 0000:00:1d.3: new USB bus registered, assigned bus number 5
hub 5-0:1.0: USB hub found
hub 5-0:1.0: 2 ports detected
usbcore: registered new driver usblp
drivers/usb/class/usblp.c: v0.13: USB Printer Device Class driver
Initializing USB Mass Storage driver...
usbcore: registered new driver usb-storage
USB Mass Storage support registered.
usbcore: registered new driver usbhid
drivers/usb/input/hid-core.c: v2.0:USB HID core driver
mice: PS/2 mouse device common for all mice
input: AT Translated Set 2 keyboard on isa0060/serio0
input: ImPS/2 Generic Wheel Mouse on isa0060/serio1
Advanced Linux Sound Architecture Driver Version 1.0.6 (Sun Aug 15 07:17:53 2004 UTC).
ACPI: PCI interrupt 0000:00:1f.5[B] -> GSI 17 (level, low) -> IRQ 17
AC'97 0 analog subsections not ready
intel8x0_measure_ac97_clock: measured 49987 usecs
intel8x0: clocking to 48000
ALSA device list:
  #0: Intel ICH5 with AD1985 at 0xfebff800, irq 17
oprofile: using NMI interrupt.
NET: Registered protocol family 2
IP: routing cache hash table of 128 buckets, 21Kbytes
TCP: Hash tables configured (established 1024 bind 1560)
ip_conntrack version 2.1 (4089 buckets, 32712 max) - 308 bytes per conntrack
ip_tables: (C) 2000-2002 Netfilter core team
ipt_recent v0.3.1: Stephen Frost <sfrost@snowman.net>.  http://snowman.net/projects/ipt_recent/
arp_tables: (C) 2002 David S. Miller
NET: Registered protocol family 1
NET: Registered protocol family 17
Starting balanced_irq
ACPI: (supports S0 S1 S3 S4 S5)
ACPI wakeup devices: 
P0P4 MC97 USB1 USB2 USB3 USB4 EUSB PS2K PS2M ILAN 
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 192k freed
[...skip...]
Warning: dev (pts0) tty->count(16) != #fd's(8) in tty_open
Warning: dev (pts0) tty->count(16) != #fd's(11) in tty_open
Warning: dev (pts0) tty->count(17) != #fd's(13) in tty_open
Warning: dev (pts0) tty->count(18) != #fd's(16) in tty_open
Warning: dev (pts0) tty->count(18) != #fd's(16) in tty_open
Warning: dev (pts0) tty->count(18) != #fd's(16) in tty_open
Warning: dev (pts0) tty->count(19) != #fd's(16) in tty_open
Warning: dev (pts0) tty->count(19) != #fd's(16) in release_dev
Warning: dev (pts0) tty->count(18) != #fd's(16) in tty_open
Warning: dev (pts0) tty->count(18) != #fd's(16) in tty_open
Warning: dev (pts0) tty->count(19) != #fd's(17) in tty_open
Warning: dev (pts0) tty->count(19) != #fd's(17) in release_dev
Warning: dev (pts0) tty->count(18) != #fd's(17) in tty_open
Warning: dev (pts0) tty->count(18) != #fd's(17) in tty_open
Warning: dev (pts0) tty->count(19) != #fd's(17) in release_dev
Warning: dev (pts0) tty->count(17) != #fd's(18) in release_dev
Warning: dev (pts0) tty->count(16) != #fd's(19) in release_dev
Warning: dev (pts0) tty->count(15) != #fd's(18) in release_dev
Warning: dev (pts0) tty->count(14) != #fd's(18) in release_dev
Warning: dev (pts0) tty->count(14) != #fd's(18) in release_dev
Warning: dev (pts0) tty->count(13) != #fd's(18) in tty_open
Warning: dev (pts0) tty->count(14) != #fd's(19) in release_dev
Warning: dev (pts0) tty->count(13) != #fd's(18) in release_dev
Warning: dev (pts0) tty->count(13) != #fd's(17) in release_dev
Warning: dev (pts0) tty->count(12) != #fd's(18) in tty_open
Warning: dev (pts0) tty->count(12) != #fd's(18) in release_dev
Warning: dev (pts0) tty->count(12) != #fd's(18) in release_dev
Warning: dev (pts0) tty->count(28) != #fd's(16) in tty_open
Warning: dev (pts0) tty->count(528) != #fd's(13) in tty_open
Warning: dev (pts0) tty->count(528) != #fd's(13) in release_dev
Warning: dev (pts0) tty->count(527) != #fd's(12) in tty_open
Warning: dev (pts0) tty->count(528) != #fd's(12) in release_dev
Warning: dev (pts0) tty->count(538) != #fd's(28) in release_dev
Warning: dev (pts0) tty->count(537) != #fd's(528) in tty_open
Warning: dev (pts0) tty->count(537) != #fd's(527) in release_dev
Warning: dev (pts0) tty->count(536) != #fd's(527) in tty_open
Warning: dev (pts0) tty->count(536) != #fd's(527) in tty_open
Warning: dev (pts0) tty->count(536) != #fd's(527) in release_dev
Warning: dev (pts0) tty->count(535) != #fd's(527) in tty_open
Warning: dev (pts0) tty->count(535) != #fd's(530) in tty_open
[...skip...]
Warning: dev (pts0) tty->count(11) != #fd's(71) in release_dev
Warning: dev (pts0) tty->count(10) != #fd's(71) in release_dev
Warning: dev (pts0) tty->count(9) != #fd's(71) in release_dev
Warning: dev (pts0) tty->count(8) != #fd's(71) in release_dev
Warning: dev (pts0) tty->count(7) != #fd's(71) in release_dev
Warning: dev (pts0) tty->count(6) != #fd's(71) in release_dev
Warning: dev (pts0) tty->count(5) != #fd's(71) in release_dev
Warning: dev (pts0) tty->count(4) != #fd's(71) in release_dev
Warning: dev (pts0) tty->count(3) != #fd's(71) in release_dev
eggcups: page allocation failure. order:1, mode:0x20
 [<c01439d8>] __alloc_pages+0x228/0x3ff (8)
 [<c0143bce>] __get_free_pages+0x1f/0x3b (68)
 [<c014719c>] kmem_getpages+0x25/0xe0 (8)
 [<c0147ee1>] cache_grow+0xca/0x16f (24)
 [<c0148067>] cache_alloc_refill+0xe1/0x230 (52)
 [<c01484a4>] __kmalloc+0x76/0x98 (48)
 [<c0325521>] pskb_expand_head+0x51/0x123 (24)
 [<c032a8ae>] skb_checksum_help+0x103/0x114 (40)
 [<c037a8d7>] ip_nat_fn+0x203/0x215 (36)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c03352bb>] nf_iterate+0x71/0xa5 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (20)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (16)
 [<c03355b9>] nf_hook_slow+0x77/0x125 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c0343bed>] dst_output+0x0/0x29 (28)
 [<c034131a>] ip_finish_output+0x24c/0x251 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (24)
 [<c0343bed>] dst_output+0x0/0x29 (24)
 [<c0343c01>] dst_output+0x14/0x29 (4)
 [<c033561e>] nf_hook_slow+0xdc/0x125 (8)
 [<c0343bed>] dst_output+0x0/0x29 (28)
 [<c0341a9e>] ip_queue_xmit+0x51d/0x61c (32)
 [<c0343bed>] dst_output+0x0/0x29 (24)
 [<c0143645>] buffered_rmqueue+0x89/0x1f4 (28)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (48)
 [<c039e226>] cond_resched+0x20/0x7f (32)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (20)
 [<c039f019>] _spin_unlock+0x12/0x2d (16)
 [<c039f019>] _spin_unlock+0x12/0x2d (16)
 [<c0147f5e>] cache_grow+0x147/0x16f (8)
 [<c035306c>] tcp_transmit_skb+0x521/0x8da (28)
 [<c0353f06>] tcp_write_xmit+0x16a/0x2c7 (72)
 [<c039e226>] cond_resched+0x20/0x7f (16)
 [<c03473fa>] tcp_sendmsg+0x541/0x1260 (40)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (20)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (20)
 [<c039f0e1>] _spin_unlock_irq+0x12/0x2e (48)
 [<c036a404>] inet_sendmsg+0x4d/0x59 (28)
 [<c0320fce>] sock_sendmsg+0xe5/0x100 (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (72)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (28)
 [<c039f019>] _spin_unlock+0x12/0x2d (40)
 [<c0133550>] autoremove_wake_function+0x0/0x57 (52)
 [<c0320d33>] sockfd_lookup+0x1c/0x77 (32)
 [<c0322424>] sys_sendto+0xe3/0xfe (20)
 [<c03221c3>] sys_connect+0x91/0xb1 (68)
 [<c0324ae2>] sock_common_setsockopt+0x36/0x3a (96)
 [<c0322476>] sys_send+0x37/0x3b (40)
 [<c0322ce9>] sys_socketcall+0x139/0x256 (28)
 [<c0105259>] sysenter_past_esp+0x52/0x71 (64)
preempt count: 00000001
. 1-level deep critical section nesting:
.. entry 1: print_traces+0x17/0x50 / (0x0)

eggcups: page allocation failure. order:1, mode:0x20
 [<c01439d8>] __alloc_pages+0x228/0x3ff (8)
 [<c0143bce>] __get_free_pages+0x1f/0x3b (68)
 [<c014719c>] kmem_getpages+0x25/0xe0 (8)
 [<c0147ee1>] cache_grow+0xca/0x16f (24)
 [<c0148067>] cache_alloc_refill+0xe1/0x230 (52)
 [<c01484a4>] __kmalloc+0x76/0x98 (48)
 [<c0325521>] pskb_expand_head+0x51/0x123 (24)
 [<c032a8ae>] skb_checksum_help+0x103/0x114 (40)
 [<c037a8d7>] ip_nat_fn+0x203/0x215 (36)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c03352bb>] nf_iterate+0x71/0xa5 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (20)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (16)
 [<c03355b9>] nf_hook_slow+0x77/0x125 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c0343bed>] dst_output+0x0/0x29 (28)
 [<c034131a>] ip_finish_output+0x24c/0x251 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (24)
 [<c0343bed>] dst_output+0x0/0x29 (24)
 [<c0343c01>] dst_output+0x14/0x29 (4)
 [<c033561e>] nf_hook_slow+0xdc/0x125 (8)
 [<c0343bed>] dst_output+0x0/0x29 (28)
 [<c0341a9e>] ip_queue_xmit+0x51d/0x61c (32)
 [<c0343bed>] dst_output+0x0/0x29 (24)
 [<c0324ee0>] kfree_skbmem+0x24/0x2c (36)
 [<c0324f81>] __kfree_skb+0x99/0xf9 (16)
 [<c034d71c>] tcp_ack_saw_tstamp+0x22/0x57 (8)
 [<c034df0b>] tcp_clean_rtx_queue+0x40b/0x433 (16)
 [<c0324610>] sk_reset_timer+0x1f/0x2f (16)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (52)
 [<c035306c>] tcp_transmit_skb+0x521/0x8da (52)
 [<c0353f06>] tcp_write_xmit+0x16a/0x2c7 (72)
 [<c039e226>] cond_resched+0x20/0x7f (16)
 [<c03473fa>] tcp_sendmsg+0x541/0x1260 (40)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (20)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (20)
 [<c039f0e1>] _spin_unlock_irq+0x12/0x2e (48)
 [<c036a404>] inet_sendmsg+0x4d/0x59 (28)
 [<c0320fce>] sock_sendmsg+0xe5/0x100 (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (72)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (28)
 [<c039f019>] _spin_unlock+0x12/0x2d (40)
 [<c0133550>] autoremove_wake_function+0x0/0x57 (52)
 [<c0320d33>] sockfd_lookup+0x1c/0x77 (32)
 [<c0322424>] sys_sendto+0xe3/0xfe (20)
 [<c03221c3>] sys_connect+0x91/0xb1 (68)
 [<c0324ae2>] sock_common_setsockopt+0x36/0x3a (96)
 [<c0322476>] sys_send+0x37/0x3b (40)
 [<c0322ce9>] sys_socketcall+0x139/0x256 (28)
 [<c0105259>] sysenter_past_esp+0x52/0x71 (64)
preempt count: 00000001
. 1-level deep critical section nesting:
.. entry 1: print_traces+0x17/0x50 / (0x0)

eggcups: page allocation failure. order:1, mode:0x20
 [<c01439d8>] __alloc_pages+0x228/0x3ff (8)
 [<c0143bce>] __get_free_pages+0x1f/0x3b (68)
 [<c014719c>] kmem_getpages+0x25/0xe0 (8)
 [<c0147ee1>] cache_grow+0xca/0x16f (24)
 [<c0148067>] cache_alloc_refill+0xe1/0x230 (52)
 [<c01484a4>] __kmalloc+0x76/0x98 (48)
 [<c0325521>] pskb_expand_head+0x51/0x123 (24)
 [<c032a8ae>] skb_checksum_help+0x103/0x114 (40)
 [<c037a8d7>] ip_nat_fn+0x203/0x215 (36)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c03352bb>] nf_iterate+0x71/0xa5 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (20)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (16)
 [<c03355b9>] nf_hook_slow+0x77/0x125 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c0343bed>] dst_output+0x0/0x29 (28)
 [<c034131a>] ip_finish_output+0x24c/0x251 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (24)
 [<c0343bed>] dst_output+0x0/0x29 (24)
 [<c0343c01>] dst_output+0x14/0x29 (4)
 [<c033561e>] nf_hook_slow+0xdc/0x125 (8)
 [<c0343bed>] dst_output+0x0/0x29 (28)
 [<c0341a9e>] ip_queue_xmit+0x51d/0x61c (32)
 [<c0343bed>] dst_output+0x0/0x29 (24)
 [<c0324ee0>] kfree_skbmem+0x24/0x2c (36)
 [<c0324f81>] __kfree_skb+0x99/0xf9 (16)
 [<c034d71c>] tcp_ack_saw_tstamp+0x22/0x57 (8)
 [<c034df0b>] tcp_clean_rtx_queue+0x40b/0x433 (16)
 [<c0324610>] sk_reset_timer+0x1f/0x2f (16)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (52)
 [<c035306c>] tcp_transmit_skb+0x521/0x8da (52)
 [<c0353f06>] tcp_write_xmit+0x16a/0x2c7 (72)
 [<c039e226>] cond_resched+0x20/0x7f (16)
 [<c03473fa>] tcp_sendmsg+0x541/0x1260 (40)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (20)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (20)
 [<c039f0e1>] _spin_unlock_irq+0x12/0x2e (48)
 [<c036a404>] inet_sendmsg+0x4d/0x59 (28)
 [<c0320fce>] sock_sendmsg+0xe5/0x100 (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (72)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (28)
 [<c039f019>] _spin_unlock+0x12/0x2d (40)
 [<c0133550>] autoremove_wake_function+0x0/0x57 (52)
 [<c0320d33>] sockfd_lookup+0x1c/0x77 (32)
 [<c0322424>] sys_sendto+0xe3/0xfe (20)
 [<c03221c3>] sys_connect+0x91/0xb1 (68)
 [<c0324ae2>] sock_common_setsockopt+0x36/0x3a (96)
 [<c0322476>] sys_send+0x37/0x3b (40)
 [<c0322ce9>] sys_socketcall+0x139/0x256 (28)
 [<c0105259>] sysenter_past_esp+0x52/0x71 (64)
preempt count: 00000001
. 1-level deep critical section nesting:
.. entry 1: print_traces+0x17/0x50 / (0x0)

eggcups: page allocation failure. order:1, mode:0x20
 [<c01439d8>] __alloc_pages+0x228/0x3ff (8)
 [<c0143bce>] __get_free_pages+0x1f/0x3b (68)
 [<c014719c>] kmem_getpages+0x25/0xe0 (8)
 [<c0147ee1>] cache_grow+0xca/0x16f (24)
 [<c0148067>] cache_alloc_refill+0xe1/0x230 (52)
 [<c01484a4>] __kmalloc+0x76/0x98 (48)
 [<c0325521>] pskb_expand_head+0x51/0x123 (24)
 [<c032a8ae>] skb_checksum_help+0x103/0x114 (40)
 [<c037a8d7>] ip_nat_fn+0x203/0x215 (36)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c03352bb>] nf_iterate+0x71/0xa5 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (20)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (16)
 [<c03355b9>] nf_hook_slow+0x77/0x125 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c0343bed>] dst_output+0x0/0x29 (28)
 [<c034131a>] ip_finish_output+0x24c/0x251 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (24)
 [<c0343bed>] dst_output+0x0/0x29 (24)
 [<c0343c01>] dst_output+0x14/0x29 (4)
 [<c033561e>] nf_hook_slow+0xdc/0x125 (8)
 [<c0343bed>] dst_output+0x0/0x29 (28)
 [<c0341a9e>] ip_queue_xmit+0x51d/0x61c (32)
 [<c0343bed>] dst_output+0x0/0x29 (24)
 [<c0324ee0>] kfree_skbmem+0x24/0x2c (36)
 [<c0324f81>] __kfree_skb+0x99/0xf9 (16)
 [<c034d71c>] tcp_ack_saw_tstamp+0x22/0x57 (8)
 [<c034df0b>] tcp_clean_rtx_queue+0x40b/0x433 (16)
 [<c0324610>] sk_reset_timer+0x1f/0x2f (16)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (52)
 [<c035306c>] tcp_transmit_skb+0x521/0x8da (52)
 [<c0353f06>] tcp_write_xmit+0x16a/0x2c7 (72)
 [<c039e226>] cond_resched+0x20/0x7f (16)
 [<c03473fa>] tcp_sendmsg+0x541/0x1260 (40)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (20)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (20)
 [<c039f0e1>] _spin_unlock_irq+0x12/0x2e (48)
 [<c036a404>] inet_sendmsg+0x4d/0x59 (28)
 [<c0320fce>] sock_sendmsg+0xe5/0x100 (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (72)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (28)
 [<c039f019>] _spin_unlock+0x12/0x2d (40)
 [<c0133550>] autoremove_wake_function+0x0/0x57 (52)
 [<c0320d33>] sockfd_lookup+0x1c/0x77 (32)
 [<c0322424>] sys_sendto+0xe3/0xfe (20)
 [<c03221c3>] sys_connect+0x91/0xb1 (68)
 [<c0324ae2>] sock_common_setsockopt+0x36/0x3a (96)
 [<c0322476>] sys_send+0x37/0x3b (40)
 [<c0322ce9>] sys_socketcall+0x139/0x256 (28)
 [<c0105259>] sysenter_past_esp+0x52/0x71 (64)
preempt count: 00000001
. 1-level deep critical section nesting:
.. entry 1: print_traces+0x17/0x50 / (0x0)

eggcups: page allocation failure. order:1, mode:0x20
 [<c01439d8>] __alloc_pages+0x228/0x3ff (8)
 [<c0143bce>] __get_free_pages+0x1f/0x3b (68)
 [<c014719c>] kmem_getpages+0x25/0xe0 (8)
 [<c0147ee1>] cache_grow+0xca/0x16f (24)
 [<c0148067>] cache_alloc_refill+0xe1/0x230 (52)
 [<c01484a4>] __kmalloc+0x76/0x98 (48)
 [<c0325521>] pskb_expand_head+0x51/0x123 (24)
 [<c032a8ae>] skb_checksum_help+0x103/0x114 (40)
 [<c037a8d7>] ip_nat_fn+0x203/0x215 (36)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c03352bb>] nf_iterate+0x71/0xa5 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (20)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (16)
 [<c03355b9>] nf_hook_slow+0x77/0x125 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c0343bed>] dst_output+0x0/0x29 (28)
 [<c034131a>] ip_finish_output+0x24c/0x251 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (24)
 [<c0343bed>] dst_output+0x0/0x29 (24)
 [<c0343c01>] dst_output+0x14/0x29 (4)
 [<c033561e>] nf_hook_slow+0xdc/0x125 (8)
 [<c0343bed>] dst_output+0x0/0x29 (28)
 [<c0341a9e>] ip_queue_xmit+0x51d/0x61c (32)
 [<c0343bed>] dst_output+0x0/0x29 (24)
 [<c0324ee0>] kfree_skbmem+0x24/0x2c (36)
 [<c0324f81>] __kfree_skb+0x99/0xf9 (16)
 [<c034d71c>] tcp_ack_saw_tstamp+0x22/0x57 (8)
 [<c034df0b>] tcp_clean_rtx_queue+0x40b/0x433 (16)
 [<c0324610>] sk_reset_timer+0x1f/0x2f (16)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (52)
 [<c035306c>] tcp_transmit_skb+0x521/0x8da (52)
 [<c0353f06>] tcp_write_xmit+0x16a/0x2c7 (72)
 [<c039e226>] cond_resched+0x20/0x7f (16)
 [<c03473fa>] tcp_sendmsg+0x541/0x1260 (40)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (20)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (20)
 [<c039f0e1>] _spin_unlock_irq+0x12/0x2e (48)
 [<c036a404>] inet_sendmsg+0x4d/0x59 (28)
 [<c0320fce>] sock_sendmsg+0xe5/0x100 (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (72)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (28)
 [<c039f019>] _spin_unlock+0x12/0x2d (40)
 [<c0133550>] autoremove_wake_function+0x0/0x57 (52)
 [<c0320d33>] sockfd_lookup+0x1c/0x77 (32)
 [<c0322424>] sys_sendto+0xe3/0xfe (20)
 [<c03221c3>] sys_connect+0x91/0xb1 (68)
 [<c0324ae2>] sock_common_setsockopt+0x36/0x3a (96)
 [<c0322476>] sys_send+0x37/0x3b (40)
 [<c0322ce9>] sys_socketcall+0x139/0x256 (28)
 [<c0105259>] sysenter_past_esp+0x52/0x71 (64)
preempt count: 00000001
. 1-level deep critical section nesting:
.. entry 1: print_traces+0x17/0x50 / (0x0)

eggcups: page allocation failure. order:1, mode:0x20
 [<c01439d8>] __alloc_pages+0x228/0x3ff (8)
 [<c0143bce>] __get_free_pages+0x1f/0x3b (68)
 [<c014719c>] kmem_getpages+0x25/0xe0 (8)
 [<c0147ee1>] cache_grow+0xca/0x16f (24)
 [<c0148067>] cache_alloc_refill+0xe1/0x230 (52)
 [<c01484a4>] __kmalloc+0x76/0x98 (48)
 [<c0325521>] pskb_expand_head+0x51/0x123 (24)
 [<c032a8ae>] skb_checksum_help+0x103/0x114 (40)
 [<c037a8d7>] ip_nat_fn+0x203/0x215 (36)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c03352bb>] nf_iterate+0x71/0xa5 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (20)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (16)
 [<c03355b9>] nf_hook_slow+0x77/0x125 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c0343bed>] dst_output+0x0/0x29 (28)
 [<c034131a>] ip_finish_output+0x24c/0x251 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (24)
 [<c0343bed>] dst_output+0x0/0x29 (24)
 [<c0343c01>] dst_output+0x14/0x29 (4)
 [<c033561e>] nf_hook_slow+0xdc/0x125 (8)
 [<c0343bed>] dst_output+0x0/0x29 (28)
 [<c0341a9e>] ip_queue_xmit+0x51d/0x61c (32)
 [<c0343bed>] dst_output+0x0/0x29 (24)
 [<c0324ee0>] kfree_skbmem+0x24/0x2c (36)
 [<c0324f81>] __kfree_skb+0x99/0xf9 (16)
 [<c034d71c>] tcp_ack_saw_tstamp+0x22/0x57 (8)
 [<c034df0b>] tcp_clean_rtx_queue+0x40b/0x433 (16)
 [<c0324610>] sk_reset_timer+0x1f/0x2f (16)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (52)
 [<c035306c>] tcp_transmit_skb+0x521/0x8da (52)
 [<c0353f06>] tcp_write_xmit+0x16a/0x2c7 (72)
 [<c039e226>] cond_resched+0x20/0x7f (16)
 [<c03473fa>] tcp_sendmsg+0x541/0x1260 (40)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (20)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (20)
 [<c039f0e1>] _spin_unlock_irq+0x12/0x2e (48)
 [<c036a404>] inet_sendmsg+0x4d/0x59 (28)
 [<c0320fce>] sock_sendmsg+0xe5/0x100 (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (72)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (28)
 [<c039f019>] _spin_unlock+0x12/0x2d (40)
 [<c0133550>] autoremove_wake_function+0x0/0x57 (52)
 [<c0320d33>] sockfd_lookup+0x1c/0x77 (32)
 [<c0322424>] sys_sendto+0xe3/0xfe (20)
 [<c03221c3>] sys_connect+0x91/0xb1 (68)
 [<c0324ae2>] sock_common_setsockopt+0x36/0x3a (96)
 [<c0322476>] sys_send+0x37/0x3b (40)
 [<c0322ce9>] sys_socketcall+0x139/0x256 (28)
 [<c0105259>] sysenter_past_esp+0x52/0x71 (64)
preempt count: 00000001
. 1-level deep critical section nesting:
.. entry 1: print_traces+0x17/0x50 / (0x0)

eggcups: page allocation failure. order:1, mode:0x20
 [<c01439d8>] __alloc_pages+0x228/0x3ff (8)
 [<c0143bce>] __get_free_pages+0x1f/0x3b (68)
 [<c014719c>] kmem_getpages+0x25/0xe0 (8)
 [<c0147ee1>] cache_grow+0xca/0x16f (24)
 [<c0148067>] cache_alloc_refill+0xe1/0x230 (52)
 [<c01484a4>] __kmalloc+0x76/0x98 (48)
 [<c0325521>] pskb_expand_head+0x51/0x123 (24)
 [<c032a8ae>] skb_checksum_help+0x103/0x114 (40)
 [<c037a8d7>] ip_nat_fn+0x203/0x215 (36)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c03352bb>] nf_iterate+0x71/0xa5 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (20)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (16)
 [<c03355b9>] nf_hook_slow+0x77/0x125 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c0343bed>] dst_output+0x0/0x29 (28)
 [<c034131a>] ip_finish_output+0x24c/0x251 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (24)
 [<c0343bed>] dst_output+0x0/0x29 (24)
 [<c0343c01>] dst_output+0x14/0x29 (4)
 [<c033561e>] nf_hook_slow+0xdc/0x125 (8)
 [<c0343bed>] dst_output+0x0/0x29 (28)
 [<c0341a9e>] ip_queue_xmit+0x51d/0x61c (32)
 [<c0343bed>] dst_output+0x0/0x29 (24)
 [<c0324ee0>] kfree_skbmem+0x24/0x2c (36)
 [<c0324f81>] __kfree_skb+0x99/0xf9 (16)
 [<c034d71c>] tcp_ack_saw_tstamp+0x22/0x57 (8)
 [<c034df0b>] tcp_clean_rtx_queue+0x40b/0x433 (16)
 [<c0324610>] sk_reset_timer+0x1f/0x2f (16)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (52)
 [<c035306c>] tcp_transmit_skb+0x521/0x8da (52)
 [<c0353f06>] tcp_write_xmit+0x16a/0x2c7 (72)
 [<c039e226>] cond_resched+0x20/0x7f (16)
 [<c03473fa>] tcp_sendmsg+0x541/0x1260 (40)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (20)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (20)
 [<c039f0e1>] _spin_unlock_irq+0x12/0x2e (48)
 [<c036a404>] inet_sendmsg+0x4d/0x59 (28)
 [<c0320fce>] sock_sendmsg+0xe5/0x100 (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (72)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (28)
 [<c039f019>] _spin_unlock+0x12/0x2d (40)
 [<c0133550>] autoremove_wake_function+0x0/0x57 (52)
 [<c0320d33>] sockfd_lookup+0x1c/0x77 (32)
 [<c0322424>] sys_sendto+0xe3/0xfe (20)
 [<c03221c3>] sys_connect+0x91/0xb1 (68)
 [<c0324ae2>] sock_common_setsockopt+0x36/0x3a (96)
 [<c0322476>] sys_send+0x37/0x3b (40)
 [<c0322ce9>] sys_socketcall+0x139/0x256 (28)
 [<c0105259>] sysenter_past_esp+0x52/0x71 (64)
preempt count: 00000001
. 1-level deep critical section nesting:
.. entry 1: print_traces+0x17/0x50 / (0x0)

eggcups: page allocation failure. order:1, mode:0x20
 [<c01439d8>] __alloc_pages+0x228/0x3ff (8)
 [<c0143bce>] __get_free_pages+0x1f/0x3b (68)
 [<c014719c>] kmem_getpages+0x25/0xe0 (8)
 [<c0147ee1>] cache_grow+0xca/0x16f (24)
 [<c0148067>] cache_alloc_refill+0xe1/0x230 (52)
 [<c01484a4>] __kmalloc+0x76/0x98 (48)
 [<c0325521>] pskb_expand_head+0x51/0x123 (24)
 [<c032a8ae>] skb_checksum_help+0x103/0x114 (40)
 [<c037a8d7>] ip_nat_fn+0x203/0x215 (36)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c03352bb>] nf_iterate+0x71/0xa5 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (20)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (16)
 [<c03355b9>] nf_hook_slow+0x77/0x125 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c0343bed>] dst_output+0x0/0x29 (28)
 [<c034131a>] ip_finish_output+0x24c/0x251 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (24)
 [<c0343bed>] dst_output+0x0/0x29 (24)
 [<c0343c01>] dst_output+0x14/0x29 (4)
 [<c033561e>] nf_hook_slow+0xdc/0x125 (8)
 [<c0343bed>] dst_output+0x0/0x29 (28)
 [<c0341a9e>] ip_queue_xmit+0x51d/0x61c (32)
 [<c0343bed>] dst_output+0x0/0x29 (24)
 [<c0324ee0>] kfree_skbmem+0x24/0x2c (36)
 [<c0324f81>] __kfree_skb+0x99/0xf9 (16)
 [<c034d71c>] tcp_ack_saw_tstamp+0x22/0x57 (8)
 [<c034df0b>] tcp_clean_rtx_queue+0x40b/0x433 (16)
 [<c0324610>] sk_reset_timer+0x1f/0x2f (16)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (52)
 [<c035306c>] tcp_transmit_skb+0x521/0x8da (52)
 [<c0353f06>] tcp_write_xmit+0x16a/0x2c7 (72)
 [<c039e226>] cond_resched+0x20/0x7f (16)
 [<c03473fa>] tcp_sendmsg+0x541/0x1260 (40)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (20)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (20)
 [<c039f0e1>] _spin_unlock_irq+0x12/0x2e (48)
 [<c036a404>] inet_sendmsg+0x4d/0x59 (28)
 [<c0320fce>] sock_sendmsg+0xe5/0x100 (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (72)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (28)
 [<c039f019>] _spin_unlock+0x12/0x2d (40)
 [<c0133550>] autoremove_wake_function+0x0/0x57 (52)
 [<c0320d33>] sockfd_lookup+0x1c/0x77 (32)
 [<c0322424>] sys_sendto+0xe3/0xfe (20)
 [<c03221c3>] sys_connect+0x91/0xb1 (68)
 [<c0324ae2>] sock_common_setsockopt+0x36/0x3a (96)
 [<c0322476>] sys_send+0x37/0x3b (40)
 [<c0322ce9>] sys_socketcall+0x139/0x256 (28)
 [<c0105259>] sysenter_past_esp+0x52/0x71 (64)
preempt count: 00000001
. 1-level deep critical section nesting:
.. entry 1: print_traces+0x17/0x50 / (0x0)

eggcups: page allocation failure. order:1, mode:0x20
 [<c01439d8>] __alloc_pages+0x228/0x3ff (8)
 [<c0143bce>] __get_free_pages+0x1f/0x3b (68)
 [<c014719c>] kmem_getpages+0x25/0xe0 (8)
 [<c0147ee1>] cache_grow+0xca/0x16f (24)
 [<c0148067>] cache_alloc_refill+0xe1/0x230 (52)
 [<c01484a4>] __kmalloc+0x76/0x98 (48)
 [<c0325521>] pskb_expand_head+0x51/0x123 (24)
 [<c032a8ae>] skb_checksum_help+0x103/0x114 (40)
 [<c037a8d7>] ip_nat_fn+0x203/0x215 (36)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c03352bb>] nf_iterate+0x71/0xa5 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (20)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (16)
 [<c03355b9>] nf_hook_slow+0x77/0x125 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c0343bed>] dst_output+0x0/0x29 (28)
 [<c034131a>] ip_finish_output+0x24c/0x251 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (24)
 [<c0343bed>] dst_output+0x0/0x29 (24)
 [<c0343c01>] dst_output+0x14/0x29 (4)
 [<c033561e>] nf_hook_slow+0xdc/0x125 (8)
 [<c0343bed>] dst_output+0x0/0x29 (28)
 [<c0341a9e>] ip_queue_xmit+0x51d/0x61c (32)
 [<c0343bed>] dst_output+0x0/0x29 (24)
 [<c0324ee0>] kfree_skbmem+0x24/0x2c (36)
 [<c0324f81>] __kfree_skb+0x99/0xf9 (16)
 [<c034d71c>] tcp_ack_saw_tstamp+0x22/0x57 (8)
 [<c034df0b>] tcp_clean_rtx_queue+0x40b/0x433 (16)
 [<c0324610>] sk_reset_timer+0x1f/0x2f (16)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (52)
 [<c035306c>] tcp_transmit_skb+0x521/0x8da (52)
 [<c0353f06>] tcp_write_xmit+0x16a/0x2c7 (72)
 [<c039e226>] cond_resched+0x20/0x7f (16)
 [<c03473fa>] tcp_sendmsg+0x541/0x1260 (40)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (20)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (20)
 [<c039f0e1>] _spin_unlock_irq+0x12/0x2e (48)
 [<c036a404>] inet_sendmsg+0x4d/0x59 (28)
 [<c0320fce>] sock_sendmsg+0xe5/0x100 (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (72)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (28)
 [<c039f019>] _spin_unlock+0x12/0x2d (40)
 [<c0133550>] autoremove_wake_function+0x0/0x57 (52)
 [<c0320d33>] sockfd_lookup+0x1c/0x77 (32)
 [<c0322424>] sys_sendto+0xe3/0xfe (20)
 [<c03221c3>] sys_connect+0x91/0xb1 (68)
 [<c0324ae2>] sock_common_setsockopt+0x36/0x3a (96)
 [<c0322476>] sys_send+0x37/0x3b (40)
 [<c0322ce9>] sys_socketcall+0x139/0x256 (28)
 [<c0105259>] sysenter_past_esp+0x52/0x71 (64)
preempt count: 00000001
. 1-level deep critical section nesting:
.. entry 1: print_traces+0x17/0x50 / (0x0)

eggcups: page allocation failure. order:1, mode:0x20
 [<c01439d8>] __alloc_pages+0x228/0x3ff (8)
 [<c0143bce>] __get_free_pages+0x1f/0x3b (68)
 [<c014719c>] kmem_getpages+0x25/0xe0 (8)
 [<c0147ee1>] cache_grow+0xca/0x16f (24)
 [<c0148067>] cache_alloc_refill+0xe1/0x230 (52)
 [<c01484a4>] __kmalloc+0x76/0x98 (48)
 [<c0325521>] pskb_expand_head+0x51/0x123 (24)
 [<c032a8ae>] skb_checksum_help+0x103/0x114 (40)
 [<c037a8d7>] ip_nat_fn+0x203/0x215 (36)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c03352bb>] nf_iterate+0x71/0xa5 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (20)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (16)
 [<c03355b9>] nf_hook_slow+0x77/0x125 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (28)
 [<c0343bed>] dst_output+0x0/0x29 (28)
 [<c034131a>] ip_finish_output+0x24c/0x251 (4)
 [<c0343c16>] ip_finish_output2+0x0/0x1fa (24)
 [<c0343bed>] dst_output+0x0/0x29 (24)
 [<c0343c01>] dst_output+0x14/0x29 (4)
 [<c033561e>] nf_hook_slow+0xdc/0x125 (8)
 [<c0343bed>] dst_output+0x0/0x29 (28)
 [<c0341a9e>] ip_queue_xmit+0x51d/0x61c (32)
 [<c0343bed>] dst_output+0x0/0x29 (24)
 [<c0324ee0>] kfree_skbmem+0x24/0x2c (36)
 [<c0324f81>] __kfree_skb+0x99/0xf9 (16)
 [<c034d71c>] tcp_ack_saw_tstamp+0x22/0x57 (8)
 [<c034df0b>] tcp_clean_rtx_queue+0x40b/0x433 (16)
 [<c0324610>] sk_reset_timer+0x1f/0x2f (16)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (52)
 [<c035306c>] tcp_transmit_skb+0x521/0x8da (52)
 [<c0353f06>] tcp_write_xmit+0x16a/0x2c7 (72)
 [<c039e226>] cond_resched+0x20/0x7f (16)
 [<c03473fa>] tcp_sendmsg+0x541/0x1260 (40)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (20)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (20)
 [<c039f0e1>] _spin_unlock_irq+0x12/0x2e (48)
 [<c036a404>] inet_sendmsg+0x4d/0x59 (28)
 [<c0320fce>] sock_sendmsg+0xe5/0x100 (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (72)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (28)
 [<c039f019>] _spin_unlock+0x12/0x2d (40)
 [<c0133550>] autoremove_wake_function+0x0/0x57 (52)
 [<c0320d33>] sockfd_lookup+0x1c/0x77 (32)
 [<c0322424>] sys_sendto+0xe3/0xfe (20)
 [<c03221c3>] sys_connect+0x91/0xb1 (68)
 [<c0324ae2>] sock_common_setsockopt+0x36/0x3a (96)
 [<c0322476>] sys_send+0x37/0x3b (40)
 [<c0322ce9>] sys_socketcall+0x139/0x256 (28)
 [<c0105259>] sysenter_past_esp+0x52/0x71 (64)
preempt count: 00000001
. 1-level deep critical section nesting:
.. entry 1: print_traces+0x17/0x50 / (0x0)

Warning: trace overflow for c0414d40 (32), increase RWSEM_MAX_OWNERS
... disabling semaphore tracing and deadlock detection.
BUG: sleeping function called from invalid context bash(24154) at drivers/block/ll_rw_blk.c:1283
in_atomic():1 [00000001], irqs_disabled():1
 [<c011c4c7>] __might_sleep+0xbb/0xc9 (8)
 [<c026d24f>] generic_unplug_device+0x1f/0x50 (36)
 [<c026d29b>] blk_backing_dev_unplug+0x1b/0x1d (16)
 [<c01558bc>] swap_unplug_io_fn+0x58/0xb5 (8)
 [<c0162583>] block_sync_page+0x4f/0x58 (36)
 [<c013e5c4>] sync_page+0x50/0x59 (12)
 [<c039e5c0>] __wait_on_bit_lock+0x53/0x61 (8)
 [<c013e574>] sync_page+0x0/0x59 (8)
 [<c013ed4b>] __lock_page+0xa3/0xab (20)
 [<c01335a7>] wake_bit_function+0x0/0x55 (24)
 [<c01335a7>] wake_bit_function+0x0/0x55 (32)
 [<c014e9ce>] do_swap_page+0x219/0x2da (20)
 [<c014f175>] handle_mm_fault+0xdf/0x18e (52)
 [<c0117355>] do_page_fault+0x1be/0x595 (48)
 [<c039f019>] _spin_unlock+0x12/0x2d (60)
 [<c015d40e>] vfs_read+0xbf/0x12a (88)
 [<c015d6df>] sys_read+0x51/0x80 (44)
 [<c0117197>] do_page_fault+0x0/0x595 (12)
 [<c0105d3d>] error_code+0x2d/0x38 (8)
 [<c010521f>] sysenter_past_esp+0x18/0x71 (52)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: print_traces+0x17/0x50 / (0x0)

BUG: scheduling with irqs disabled: bash/0x04000001/24154
caller is cond_resched+0x5e/0x7f
 [<c039d99f>] schedule+0x6c/0xb9 (8)
 [<c039e264>] cond_resched+0x5e/0x7f (8)
 [<c039e264>] cond_resched+0x5e/0x7f (24)
 [<c026d254>] generic_unplug_device+0x24/0x50 (16)
 [<c026d29b>] blk_backing_dev_unplug+0x1b/0x1d (16)
 [<c01558bc>] swap_unplug_io_fn+0x58/0xb5 (8)
 [<c0162583>] block_sync_page+0x4f/0x58 (36)
 [<c013e5c4>] sync_page+0x50/0x59 (12)
 [<c039e5c0>] __wait_on_bit_lock+0x53/0x61 (8)
 [<c013e574>] sync_page+0x0/0x59 (8)
 [<c013ed4b>] __lock_page+0xa3/0xab (20)
 [<c01335a7>] wake_bit_function+0x0/0x55 (24)
 [<c01335a7>] wake_bit_function+0x0/0x55 (32)
 [<c014e9ce>] do_swap_page+0x219/0x2da (20)
 [<c014f175>] handle_mm_fault+0xdf/0x18e (52)
 [<c0117355>] do_page_fault+0x1be/0x595 (48)
 [<c039f019>] _spin_unlock+0x12/0x2d (60)
 [<c015d40e>] vfs_read+0xbf/0x12a (88)
 [<c015d6df>] sys_read+0x51/0x80 (44)
 [<c0117197>] do_page_fault+0x0/0x595 (12)
 [<c0105d3d>] error_code+0x2d/0x38 (8)
 [<c010521f>] sysenter_past_esp+0x18/0x71 (52)
preempt count: 04000002
. 2-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: print_traces+0x17/0x50 / (0x0)

BUG: scheduling while atomic: bash/0x04000001/24154
caller is cond_resched+0x5e/0x7f
 [<c039d8e3>] __sched_text_start+0xbef/0xc3f (8)
 [<c039e264>] cond_resched+0x5e/0x7f (8)
 [<c011f16e>] vprintk+0x11e/0x15b (12)
 [<c0106116>] dump_stack+0x1c/0x20 (56)
 [<c039d99f>] schedule+0x6c/0xb9 (16)
 [<c039e264>] cond_resched+0x5e/0x7f (8)
 [<c039e264>] cond_resched+0x5e/0x7f (24)
 [<c026d254>] generic_unplug_device+0x24/0x50 (16)
 [<c026d29b>] blk_backing_dev_unplug+0x1b/0x1d (16)
 [<c01558bc>] swap_unplug_io_fn+0x58/0xb5 (8)
 [<c0162583>] block_sync_page+0x4f/0x58 (36)
 [<c013e5c4>] sync_page+0x50/0x59 (12)
 [<c039e5c0>] __wait_on_bit_lock+0x53/0x61 (8)
 [<c013e574>] sync_page+0x0/0x59 (8)
 [<c013ed4b>] __lock_page+0xa3/0xab (20)
 [<c01335a7>] wake_bit_function+0x0/0x55 (24)
 [<c01335a7>] wake_bit_function+0x0/0x55 (32)
 [<c014e9ce>] do_swap_page+0x219/0x2da (20)
 [<c014f175>] handle_mm_fault+0xdf/0x18e (52)
 [<c0117355>] do_page_fault+0x1be/0x595 (48)
 [<c039f019>] _spin_unlock+0x12/0x2d (60)
 [<c015d40e>] vfs_read+0xbf/0x12a (88)
 [<c015d6df>] sys_read+0x51/0x80 (44)
 [<c0117197>] do_page_fault+0x0/0x595 (12)
 [<c0105d3d>] error_code+0x2d/0x38 (8)
 [<c010521f>] sysenter_past_esp+0x18/0x71 (52)
preempt count: 04000002
. 2-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: print_traces+0x17/0x50 / (0x0)

BUG: scheduling while atomic: bash/0x00000001/24154
caller is down_write+0x14d/0x17c
 [<c039d8e3>] __sched_text_start+0xbef/0xc3f (8)
 [<c039e946>] down_write+0x14d/0x17c (8)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (16)
 [<c039e946>] down_write+0x14d/0x17c (52)
 [<c026d262>] generic_unplug_device+0x32/0x50 (40)
 [<c026d29b>] blk_backing_dev_unplug+0x1b/0x1d (16)
 [<c01558bc>] swap_unplug_io_fn+0x58/0xb5 (8)
 [<c0162583>] block_sync_page+0x4f/0x58 (36)
 [<c013e5c4>] sync_page+0x50/0x59 (12)
 [<c039e5c0>] __wait_on_bit_lock+0x53/0x61 (8)
 [<c013e574>] sync_page+0x0/0x59 (8)
 [<c013ed4b>] __lock_page+0xa3/0xab (20)
 [<c01335a7>] wake_bit_function+0x0/0x55 (24)
 [<c01335a7>] wake_bit_function+0x0/0x55 (32)
 [<c014e9ce>] do_swap_page+0x219/0x2da (20)
 [<c014f175>] handle_mm_fault+0xdf/0x18e (52)
 [<c0117355>] do_page_fault+0x1be/0x595 (48)
 [<c039f019>] _spin_unlock+0x12/0x2d (60)
 [<c015d40e>] vfs_read+0xbf/0x12a (88)
 [<c015d6df>] sys_read+0x51/0x80 (44)
 [<c0117197>] do_page_fault+0x0/0x595 (12)
 [<c0105d3d>] error_code+0x2d/0x38 (8)
 [<c010521f>] sysenter_past_esp+0x18/0x71 (52)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: print_traces+0x17/0x50 / (0x0)

BUG: scheduling while atomic: bash/0x00000001/24154
caller is io_schedule+0x26/0x30
 [<c039d8e3>] __sched_text_start+0xbef/0xc3f (8)
 [<c039e2c8>] io_schedule+0x26/0x30 (8)
 [<c039e2c8>] io_schedule+0x26/0x30 (116)
 [<c013e5b8>] sync_page+0x44/0x59 (12)
 [<c039e5c0>] __wait_on_bit_lock+0x53/0x61 (8)
 [<c013e574>] sync_page+0x0/0x59 (8)
 [<c013ed4b>] __lock_page+0xa3/0xab (20)
 [<c01335a7>] wake_bit_function+0x0/0x55 (24)
 [<c01335a7>] wake_bit_function+0x0/0x55 (32)
 [<c014e9ce>] do_swap_page+0x219/0x2da (20)
 [<c014f175>] handle_mm_fault+0xdf/0x18e (52)
 [<c0117355>] do_page_fault+0x1be/0x595 (48)
 [<c039f019>] _spin_unlock+0x12/0x2d (60)
 [<c015d40e>] vfs_read+0xbf/0x12a (88)
 [<c015d6df>] sys_read+0x51/0x80 (44)
 [<c0117197>] do_page_fault+0x0/0x595 (12)
 [<c0105d3d>] error_code+0x2d/0x38 (8)
 [<c010521f>] sysenter_past_esp+0x18/0x71 (52)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: print_traces+0x17/0x50 / (0x0)

BUG: sleeping function called from invalid context bash(24154) at lib/rwsem-generic.c:319
in_atomic():1 [00000001], irqs_disabled():0
 [<c011c4c7>] __might_sleep+0xbb/0xc9 (8)
 [<c039e69d>] down_read+0x2e/0x18a (36)
 [<c015587d>] swap_unplug_io_fn+0x19/0xb5 (40)
 [<c0162583>] block_sync_page+0x4f/0x58 (36)
 [<c013e5c4>] sync_page+0x50/0x59 (12)
 [<c039e5c0>] __wait_on_bit_lock+0x53/0x61 (8)
 [<c013e574>] sync_page+0x0/0x59 (8)
 [<c013ed4b>] __lock_page+0xa3/0xab (20)
 [<c01335a7>] wake_bit_function+0x0/0x55 (24)
 [<c01335a7>] wake_bit_function+0x0/0x55 (32)
 [<c014e9ce>] do_swap_page+0x219/0x2da (20)
 [<c014f175>] handle_mm_fault+0xdf/0x18e (52)
 [<c0117355>] do_page_fault+0x1be/0x595 (48)
 [<c039f019>] _spin_unlock+0x12/0x2d (60)
 [<c015d40e>] vfs_read+0xbf/0x12a (88)
 [<c015d6df>] sys_read+0x51/0x80 (44)
 [<c0117197>] do_page_fault+0x0/0x595 (12)
 [<c0105d3d>] error_code+0x2d/0x38 (8)
 [<c010521f>] sysenter_past_esp+0x18/0x71 (52)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: print_traces+0x17/0x50 / (0x0)

BUG: scheduling while atomic: bash/0x04000001/24154
caller is cond_resched+0x5e/0x7f
 [<c039d8e3>] __sched_text_start+0xbef/0xc3f (8)
 [<c039e264>] cond_resched+0x5e/0x7f (8)
 [<c011f16e>] vprintk+0x11e/0x15b (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (16)
 [<c0106116>] dump_stack+0x1c/0x20 (40)
 [<c039e264>] cond_resched+0x5e/0x7f (36)
 [<c039e6a2>] down_read+0x33/0x18a (16)
 [<c015587d>] swap_unplug_io_fn+0x19/0xb5 (40)
 [<c0162583>] block_sync_page+0x4f/0x58 (36)
 [<c013e5c4>] sync_page+0x50/0x59 (12)
 [<c039e5c0>] __wait_on_bit_lock+0x53/0x61 (8)
 [<c013e574>] sync_page+0x0/0x59 (8)
 [<c013ed4b>] __lock_page+0xa3/0xab (20)
 [<c01335a7>] wake_bit_function+0x0/0x55 (24)
 [<c01335a7>] wake_bit_function+0x0/0x55 (32)
 [<c014e9ce>] do_swap_page+0x219/0x2da (20)
 [<c014f175>] handle_mm_fault+0xdf/0x18e (52)
 [<c0117355>] do_page_fault+0x1be/0x595 (48)
 [<c039f019>] _spin_unlock+0x12/0x2d (60)
 [<c015d40e>] vfs_read+0xbf/0x12a (88)
 [<c015d6df>] sys_read+0x51/0x80 (44)
 [<c0117197>] do_page_fault+0x0/0x595 (12)
 [<c0105d3d>] error_code+0x2d/0x38 (8)
 [<c010521f>] sysenter_past_esp+0x18/0x71 (52)
preempt count: 04000002
. 2-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: print_traces+0x17/0x50 / (0x0)

BUG: scheduling while atomic: bash/0x00000001/24154
caller is down_write+0x14d/0x17c
 [<c039d8e3>] __sched_text_start+0xbef/0xc3f (8)
 [<c039e946>] down_write+0x14d/0x17c (8)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (16)
 [<c039e946>] down_write+0x14d/0x17c (52)
 [<c026d262>] generic_unplug_device+0x32/0x50 (40)
 [<c026d29b>] blk_backing_dev_unplug+0x1b/0x1d (16)
 [<c01558bc>] swap_unplug_io_fn+0x58/0xb5 (8)
 [<c0162583>] block_sync_page+0x4f/0x58 (36)
 [<c013e5c4>] sync_page+0x50/0x59 (12)
 [<c039e5c0>] __wait_on_bit_lock+0x53/0x61 (8)
 [<c013e574>] sync_page+0x0/0x59 (8)
 [<c013ed4b>] __lock_page+0xa3/0xab (20)
 [<c01335a7>] wake_bit_function+0x0/0x55 (24)
 [<c01335a7>] wake_bit_function+0x0/0x55 (32)
 [<c014e9ce>] do_swap_page+0x219/0x2da (20)
 [<c014f175>] handle_mm_fault+0xdf/0x18e (52)
 [<c0117355>] do_page_fault+0x1be/0x595 (48)
 [<c039f019>] _spin_unlock+0x12/0x2d (60)
 [<c015d40e>] vfs_read+0xbf/0x12a (88)
 [<c015d6df>] sys_read+0x51/0x80 (44)
 [<c0117197>] do_page_fault+0x0/0x595 (12)
 [<c0105d3d>] error_code+0x2d/0x38 (8)
 [<c010521f>] sysenter_past_esp+0x18/0x71 (52)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: print_traces+0x17/0x50 / (0x0)

BUG: scheduling while atomic: bash/0x00000001/24154
caller is io_schedule+0x26/0x30
 [<c039d8e3>] __sched_text_start+0xbef/0xc3f (8)
 [<c039e2c8>] io_schedule+0x26/0x30 (8)
 [<c039e2c8>] io_schedule+0x26/0x30 (116)
 [<c013e5b8>] sync_page+0x44/0x59 (12)
 [<c039e5c0>] __wait_on_bit_lock+0x53/0x61 (8)
 [<c013e574>] sync_page+0x0/0x59 (8)
 [<c013ed4b>] __lock_page+0xa3/0xab (20)
 [<c01335a7>] wake_bit_function+0x0/0x55 (24)
 [<c01335a7>] wake_bit_function+0x0/0x55 (32)
 [<c014e9ce>] do_swap_page+0x219/0x2da (20)
 [<c014f175>] handle_mm_fault+0xdf/0x18e (52)
 [<c0117355>] do_page_fault+0x1be/0x595 (48)
 [<c039f019>] _spin_unlock+0x12/0x2d (60)
 [<c015d40e>] vfs_read+0xbf/0x12a (88)
 [<c015d6df>] sys_read+0x51/0x80 (44)
 [<c0117197>] do_page_fault+0x0/0x595 (12)
 [<c0105d3d>] error_code+0x2d/0x38 (8)
 [<c010521f>] sysenter_past_esp+0x18/0x71 (52)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: print_traces+0x17/0x50 / (0x0)

BUG: scheduling while atomic: bash/0x00000001/24154
caller is down_write+0x14d/0x17c
 [<c039d8e3>] __sched_text_start+0xbef/0xc3f (8)
 [<c039e946>] down_write+0x14d/0x17c (8)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (16)
 [<c039e946>] down_write+0x14d/0x17c (52)
 [<c026d262>] generic_unplug_device+0x32/0x50 (40)
 [<c026d29b>] blk_backing_dev_unplug+0x1b/0x1d (16)
 [<c01558bc>] swap_unplug_io_fn+0x58/0xb5 (8)
 [<c0162583>] block_sync_page+0x4f/0x58 (36)
 [<c013e5c4>] sync_page+0x50/0x59 (12)
 [<c039e5c0>] __wait_on_bit_lock+0x53/0x61 (8)
 [<c013e574>] sync_page+0x0/0x59 (8)
 [<c013ed4b>] __lock_page+0xa3/0xab (20)
 [<c01335a7>] wake_bit_function+0x0/0x55 (24)
 [<c01335a7>] wake_bit_function+0x0/0x55 (32)
 [<c014e9ce>] do_swap_page+0x219/0x2da (20)
 [<c014f175>] handle_mm_fault+0xdf/0x18e (52)
 [<c0117355>] do_page_fault+0x1be/0x595 (48)
 [<c039f019>] _spin_unlock+0x12/0x2d (60)
 [<c015d40e>] vfs_read+0xbf/0x12a (88)
 [<c015d6df>] sys_read+0x51/0x80 (44)
 [<c0117197>] do_page_fault+0x0/0x595 (12)
 [<c0105d3d>] error_code+0x2d/0x38 (8)
 [<c010521f>] sysenter_past_esp+0x18/0x71 (52)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: print_traces+0x17/0x50 / (0x0)

BUG: scheduling while atomic: bash/0x00000001/24154
caller is io_schedule+0x26/0x30
 [<c039d8e3>] __sched_text_start+0xbef/0xc3f (8)
 [<c039e2c8>] io_schedule+0x26/0x30 (8)
 [<c039e2c8>] io_schedule+0x26/0x30 (116)
 [<c013e5b8>] sync_page+0x44/0x59 (12)
 [<c039e5c0>] __wait_on_bit_lock+0x53/0x61 (8)
 [<c013e574>] sync_page+0x0/0x59 (8)
 [<c013ed4b>] __lock_page+0xa3/0xab (20)
 [<c01335a7>] wake_bit_function+0x0/0x55 (24)
 [<c01335a7>] wake_bit_function+0x0/0x55 (32)
 [<c014e9ce>] do_swap_page+0x219/0x2da (20)
 [<c014f175>] handle_mm_fault+0xdf/0x18e (52)
 [<c0117355>] do_page_fault+0x1be/0x595 (48)
 [<c039f019>] _spin_unlock+0x12/0x2d (60)
 [<c015d40e>] vfs_read+0xbf/0x12a (88)
 [<c015d6df>] sys_read+0x51/0x80 (44)
 [<c0117197>] do_page_fault+0x0/0x595 (12)
 [<c0105d3d>] error_code+0x2d/0x38 (8)
 [<c010521f>] sysenter_past_esp+0x18/0x71 (52)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: print_traces+0x17/0x50 / (0x0)

BUG: sleeping function called from invalid context bash(24154) at lib/rwsem-generic.c:319
in_atomic():1 [00000001], irqs_disabled():0
 [<c011c4c7>] __might_sleep+0xbb/0xc9 (8)
 [<c039e69d>] down_read+0x2e/0x18a (36)
 [<c015587d>] swap_unplug_io_fn+0x19/0xb5 (40)
 [<c0162583>] block_sync_page+0x4f/0x58 (36)
 [<c013e5c4>] sync_page+0x50/0x59 (12)
 [<c039e5c0>] __wait_on_bit_lock+0x53/0x61 (8)
 [<c013e574>] sync_page+0x0/0x59 (8)
 [<c013ed4b>] __lock_page+0xa3/0xab (20)
 [<c01335a7>] wake_bit_function+0x0/0x55 (24)
 [<c01335a7>] wake_bit_function+0x0/0x55 (32)
 [<c014e9ce>] do_swap_page+0x219/0x2da (20)
 [<c014f175>] handle_mm_fault+0xdf/0x18e (52)
 [<c0117355>] do_page_fault+0x1be/0x595 (48)
 [<c039f019>] _spin_unlock+0x12/0x2d (60)
 [<c015d40e>] vfs_read+0xbf/0x12a (88)
 [<c015d6df>] sys_read+0x51/0x80 (44)
 [<c0117197>] do_page_fault+0x0/0x595 (12)
 [<c0105d3d>] error_code+0x2d/0x38 (8)
 [<c010521f>] sysenter_past_esp+0x18/0x71 (52)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: print_traces+0x17/0x50 / (0x0)

BUG: scheduling while atomic: bash/0x04000001/24154
caller is cond_resched+0x5e/0x7f
 [<c039d8e3>] __sched_text_start+0xbef/0xc3f (8)
 [<c039e264>] cond_resched+0x5e/0x7f (8)
 [<c011f16e>] vprintk+0x11e/0x15b (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (16)
 [<c0106116>] dump_stack+0x1c/0x20 (40)
 [<c039e264>] cond_resched+0x5e/0x7f (36)
 [<c039e6a2>] down_read+0x33/0x18a (16)
 [<c015587d>] swap_unplug_io_fn+0x19/0xb5 (40)
 [<c0162583>] block_sync_page+0x4f/0x58 (36)
 [<c013e5c4>] sync_page+0x50/0x59 (12)
 [<c039e5c0>] __wait_on_bit_lock+0x53/0x61 (8)
 [<c013e574>] sync_page+0x0/0x59 (8)
 [<c013ed4b>] __lock_page+0xa3/0xab (20)
 [<c01335a7>] wake_bit_function+0x0/0x55 (24)
 [<c01335a7>] wake_bit_function+0x0/0x55 (32)
 [<c014e9ce>] do_swap_page+0x219/0x2da (20)
 [<c014f175>] handle_mm_fault+0xdf/0x18e (52)
 [<c0117355>] do_page_fault+0x1be/0x595 (48)
 [<c039f019>] _spin_unlock+0x12/0x2d (60)
 [<c015d40e>] vfs_read+0xbf/0x12a (88)
 [<c015d6df>] sys_read+0x51/0x80 (44)
 [<c0117197>] do_page_fault+0x0/0x595 (12)
 [<c0105d3d>] error_code+0x2d/0x38 (8)
 [<c010521f>] sysenter_past_esp+0x18/0x71 (52)
preempt count: 04000002
. 2-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: print_traces+0x17/0x50 / (0x0)

BUG: scheduling while atomic: bash/0x00000001/24154
caller is down_write+0x14d/0x17c
 [<c039d8e3>] __sched_text_start+0xbef/0xc3f (8)
 [<c039e946>] down_write+0x14d/0x17c (8)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (16)
 [<c039e946>] down_write+0x14d/0x17c (52)
 [<c026d262>] generic_unplug_device+0x32/0x50 (40)
 [<c026d29b>] blk_backing_dev_unplug+0x1b/0x1d (16)
 [<c01558bc>] swap_unplug_io_fn+0x58/0xb5 (8)
 [<c0162583>] block_sync_page+0x4f/0x58 (36)
 [<c013e5c4>] sync_page+0x50/0x59 (12)
 [<c039e5c0>] __wait_on_bit_lock+0x53/0x61 (8)
 [<c013e574>] sync_page+0x0/0x59 (8)
 [<c013ed4b>] __lock_page+0xa3/0xab (20)
 [<c01335a7>] wake_bit_function+0x0/0x55 (24)
 [<c01335a7>] wake_bit_function+0x0/0x55 (32)
 [<c014e9ce>] do_swap_page+0x219/0x2da (20)
 [<c014f175>] handle_mm_fault+0xdf/0x18e (52)
 [<c0117355>] do_page_fault+0x1be/0x595 (48)
 [<c039f019>] _spin_unlock+0x12/0x2d (60)
 [<c015d40e>] vfs_read+0xbf/0x12a (88)
 [<c015d6df>] sys_read+0x51/0x80 (44)
 [<c0117197>] do_page_fault+0x0/0x595 (12)
 [<c0105d3d>] error_code+0x2d/0x38 (8)
 [<c010521f>] sysenter_past_esp+0x18/0x71 (52)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: print_traces+0x17/0x50 / (0x0)

BUG: scheduling while atomic: bash/0x00000001/24154
caller is io_schedule+0x26/0x30
 [<c039d8e3>] __sched_text_start+0xbef/0xc3f (8)
 [<c039e2c8>] io_schedule+0x26/0x30 (8)
 [<c039e2c8>] io_schedule+0x26/0x30 (116)
 [<c013e5b8>] sync_page+0x44/0x59 (12)
 [<c039e5c0>] __wait_on_bit_lock+0x53/0x61 (8)
 [<c013e574>] sync_page+0x0/0x59 (8)
 [<c013ed4b>] __lock_page+0xa3/0xab (20)
 [<c01335a7>] wake_bit_function+0x0/0x55 (24)
 [<c01335a7>] wake_bit_function+0x0/0x55 (32)
 [<c014e9ce>] do_swap_page+0x219/0x2da (20)
 [<c014f175>] handle_mm_fault+0xdf/0x18e (52)
 [<c0117355>] do_page_fault+0x1be/0x595 (48)
 [<c039f019>] _spin_unlock+0x12/0x2d (60)
 [<c015d40e>] vfs_read+0xbf/0x12a (88)
 [<c015d6df>] sys_read+0x51/0x80 (44)
 [<c0117197>] do_page_fault+0x0/0x595 (12)
 [<c0105d3d>] error_code+0x2d/0x38 (8)
 [<c010521f>] sysenter_past_esp+0x18/0x71 (52)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: print_traces+0x17/0x50 / (0x0)

BUG: scheduling while atomic: bash/0x00000001/24154
caller is down_write+0x14d/0x17c
 [<c039d8e3>] __sched_text_start+0xbef/0xc3f (8)
 [<c039e946>] down_write+0x14d/0x17c (8)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (16)
 [<c039e946>] down_write+0x14d/0x17c (52)
 [<c026d262>] generic_unplug_device+0x32/0x50 (40)
 [<c026d29b>] blk_backing_dev_unplug+0x1b/0x1d (16)
 [<c01558bc>] swap_unplug_io_fn+0x58/0xb5 (8)
 [<c0162583>] block_sync_page+0x4f/0x58 (36)
 [<c013e5c4>] sync_page+0x50/0x59 (12)
 [<c039e5c0>] __wait_on_bit_lock+0x53/0x61 (8)
 [<c013e574>] sync_page+0x0/0x59 (8)
 [<c013ed4b>] __lock_page+0xa3/0xab (20)
 [<c01335a7>] wake_bit_function+0x0/0x55 (24)
 [<c01335a7>] wake_bit_function+0x0/0x55 (32)
 [<c014e9ce>] do_swap_page+0x219/0x2da (20)
 [<c014f175>] handle_mm_fault+0xdf/0x18e (52)
 [<c0117355>] do_page_fault+0x1be/0x595 (48)
 [<c039f019>] _spin_unlock+0x12/0x2d (60)
 [<c015d40e>] vfs_read+0xbf/0x12a (88)
 [<c015d6df>] sys_read+0x51/0x80 (44)
 [<c0117197>] do_page_fault+0x0/0x595 (12)
 [<c0105d3d>] error_code+0x2d/0x38 (8)
 [<c010521f>] sysenter_past_esp+0x18/0x71 (52)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: print_traces+0x17/0x50 / (0x0)

BUG: scheduling while atomic: bash/0x00000001/24154
caller is io_schedule+0x26/0x30
 [<c039d8e3>] __sched_text_start+0xbef/0xc3f (8)
 [<c039e2c8>] io_schedule+0x26/0x30 (8)
 [<c039f019>] _spin_unlock+0x12/0x2d (28)
 [<c039f019>] _spin_unlock+0x12/0x2d (52)
 [<c039e2c8>] io_schedule+0x26/0x30 (36)
 [<c013e5b8>] sync_page+0x44/0x59 (12)
 [<c039e5c0>] __wait_on_bit_lock+0x53/0x61 (8)
 [<c013e574>] sync_page+0x0/0x59 (8)
 [<c013ed4b>] __lock_page+0xa3/0xab (20)
 [<c01335a7>] wake_bit_function+0x0/0x55 (24)
 [<c01335a7>] wake_bit_function+0x0/0x55 (32)
 [<c014e9ce>] do_swap_page+0x219/0x2da (20)
 [<c014f175>] handle_mm_fault+0xdf/0x18e (52)
 [<c0117355>] do_page_fault+0x1be/0x595 (48)
 [<c039f019>] _spin_unlock+0x12/0x2d (60)
 [<c015d40e>] vfs_read+0xbf/0x12a (88)
 [<c015d6df>] sys_read+0x51/0x80 (44)
 [<c0117197>] do_page_fault+0x0/0x595 (12)
 [<c0105d3d>] error_code+0x2d/0x38 (8)
 [<c010521f>] sysenter_past_esp+0x18/0x71 (52)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: print_traces+0x17/0x50 / (0x0)

BUG: sleeping function called from invalid context bash(24154) at lib/rwsem-generic.c:319
in_atomic():1 [00000001], irqs_disabled():0
 [<c011c4c7>] __might_sleep+0xbb/0xc9 (8)
 [<c039f019>] _spin_unlock+0x12/0x2d (24)
 [<c039e69d>] down_read+0x2e/0x18a (12)
 [<c015587d>] swap_unplug_io_fn+0x19/0xb5 (40)
 [<c0162583>] block_sync_page+0x4f/0x58 (36)
 [<c013e5c4>] sync_page+0x50/0x59 (12)
 [<c039e5c0>] __wait_on_bit_lock+0x53/0x61 (8)
 [<c013e574>] sync_page+0x0/0x59 (8)
 [<c013ed4b>] __lock_page+0xa3/0xab (20)
 [<c01335a7>] wake_bit_function+0x0/0x55 (24)
 [<c01335a7>] wake_bit_function+0x0/0x55 (32)
 [<c014e9ce>] do_swap_page+0x219/0x2da (20)
 [<c014f175>] handle_mm_fault+0xdf/0x18e (52)
 [<c0117355>] do_page_fault+0x1be/0x595 (48)
 [<c039f019>] _spin_unlock+0x12/0x2d (60)
 [<c015d40e>] vfs_read+0xbf/0x12a (88)
 [<c015d6df>] sys_read+0x51/0x80 (44)
 [<c0117197>] do_page_fault+0x0/0x595 (12)
 [<c0105d3d>] error_code+0x2d/0x38 (8)
 [<c010521f>] sysenter_past_esp+0x18/0x71 (52)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: print_traces+0x17/0x50 / (0x0)

BUG: scheduling while atomic: bash/0x00000001/24154
caller is down_write+0x14d/0x17c
 [<c039d8e3>] __sched_text_start+0xbef/0xc3f (8)
 [<c039e946>] down_write+0x14d/0x17c (8)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (16)
 [<c039e946>] down_write+0x14d/0x17c (52)
 [<c026d262>] generic_unplug_device+0x32/0x50 (40)
 [<c026d29b>] blk_backing_dev_unplug+0x1b/0x1d (16)
 [<c01558bc>] swap_unplug_io_fn+0x58/0xb5 (8)
 [<c0162583>] block_sync_page+0x4f/0x58 (36)
 [<c013e5c4>] sync_page+0x50/0x59 (12)
 [<c039e5c0>] __wait_on_bit_lock+0x53/0x61 (8)
 [<c013e574>] sync_page+0x0/0x59 (8)
 [<c013ed4b>] __lock_page+0xa3/0xab (20)
 [<c01335a7>] wake_bit_function+0x0/0x55 (24)
 [<c01335a7>] wake_bit_function+0x0/0x55 (32)
 [<c014e9ce>] do_swap_page+0x219/0x2da (20)
 [<c014f175>] handle_mm_fault+0xdf/0x18e (52)
 [<c0117355>] do_page_fault+0x1be/0x595 (48)
 [<c039f019>] _spin_unlock+0x12/0x2d (60)
 [<c015d40e>] vfs_read+0xbf/0x12a (88)
 [<c015d6df>] sys_read+0x51/0x80 (44)
 [<c0117197>] do_page_fault+0x0/0x595 (12)
 [<c0105d3d>] error_code+0x2d/0x38 (8)
 [<c010521f>] sysenter_past_esp+0x18/0x71 (52)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: print_traces+0x17/0x50 / (0x0)

BUG: scheduling while atomic: bash/0x00000001/24154
caller is io_schedule+0x26/0x30
 [<c039d8e3>] __sched_text_start+0xbef/0xc3f (8)
 [<c039e2c8>] io_schedule+0x26/0x30 (8)
 [<c039f019>] _spin_unlock+0x12/0x2d (28)
 [<c039f019>] _spin_unlock+0x12/0x2d (52)
 [<c039e2c8>] io_schedule+0x26/0x30 (36)
 [<c013e5b8>] sync_page+0x44/0x59 (12)
 [<c039e5c0>] __wait_on_bit_lock+0x53/0x61 (8)
 [<c013e574>] sync_page+0x0/0x59 (8)
 [<c013ed4b>] __lock_page+0xa3/0xab (20)
 [<c01335a7>] wake_bit_function+0x0/0x55 (24)
 [<c01335a7>] wake_bit_function+0x0/0x55 (32)
 [<c014e9ce>] do_swap_page+0x219/0x2da (20)
 [<c014f175>] handle_mm_fault+0xdf/0x18e (52)
 [<c0117355>] do_page_fault+0x1be/0x595 (48)
 [<c039f019>] _spin_unlock+0x12/0x2d (60)
 [<c015d40e>] vfs_read+0xbf/0x12a (88)
 [<c015d6df>] sys_read+0x51/0x80 (44)
 [<c0117197>] do_page_fault+0x0/0x595 (12)
 [<c0105d3d>] error_code+0x2d/0x38 (8)
 [<c010521f>] sysenter_past_esp+0x18/0x71 (52)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: print_traces+0x17/0x50 / (0x0)

BUG: scheduling while atomic: bash/0x00000001/24154
caller is down_write+0x14d/0x17c
 [<c039d8e3>] __sched_text_start+0xbef/0xc3f (8)
 [<c039e946>] down_write+0x14d/0x17c (8)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (16)
 [<c039e946>] down_write+0x14d/0x17c (52)
 [<c026d262>] generic_unplug_device+0x32/0x50 (40)
 [<c026d29b>] blk_backing_dev_unplug+0x1b/0x1d (16)
 [<c01558bc>] swap_unplug_io_fn+0x58/0xb5 (8)
 [<c0162583>] block_sync_page+0x4f/0x58 (36)
 [<c013e5c4>] sync_page+0x50/0x59 (12)
 [<c039e5c0>] __wait_on_bit_lock+0x53/0x61 (8)
 [<c013e574>] sync_page+0x0/0x59 (8)
 [<c013ed4b>] __lock_page+0xa3/0xab (20)
 [<c01335a7>] wake_bit_function+0x0/0x55 (24)
 [<c01335a7>] wake_bit_function+0x0/0x55 (32)
 [<c014e9ce>] do_swap_page+0x219/0x2da (20)
 [<c014f175>] handle_mm_fault+0xdf/0x18e (52)
 [<c0117355>] do_page_fault+0x1be/0x595 (48)
 [<c039f019>] _spin_unlock+0x12/0x2d (60)
 [<c015d40e>] vfs_read+0xbf/0x12a (88)
 [<c015d6df>] sys_read+0x51/0x80 (44)
 [<c0117197>] do_page_fault+0x0/0x595 (12)
 [<c0105d3d>] error_code+0x2d/0x38 (8)
 [<c010521f>] sysenter_past_esp+0x18/0x71 (52)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: print_traces+0x17/0x50 / (0x0)

BUG: scheduling while atomic: bash/0x00000001/24154
caller is io_schedule+0x26/0x30
 [<c039d8e3>] __sched_text_start+0xbef/0xc3f (8)
 [<c039e2c8>] io_schedule+0x26/0x30 (8)
 [<c039e2c8>] io_schedule+0x26/0x30 (116)
 [<c013e5b8>] sync_page+0x44/0x59 (12)
 [<c039e5c0>] __wait_on_bit_lock+0x53/0x61 (8)
 [<c013e574>] sync_page+0x0/0x59 (8)
 [<c013ed4b>] __lock_page+0xa3/0xab (20)
 [<c01335a7>] wake_bit_function+0x0/0x55 (24)
 [<c01335a7>] wake_bit_function+0x0/0x55 (32)
 [<c014e9ce>] do_swap_page+0x219/0x2da (20)
 [<c014f175>] handle_mm_fault+0xdf/0x18e (52)
 [<c0117355>] do_page_fault+0x1be/0x595 (48)
 [<c039f019>] _spin_unlock+0x12/0x2d (60)
 [<c015d40e>] vfs_read+0xbf/0x12a (88)
 [<c015d6df>] sys_read+0x51/0x80 (44)
 [<c0117197>] do_page_fault+0x0/0x595 (12)
 [<c0105d3d>] error_code+0x2d/0x38 (8)
 [<c010521f>] sysenter_past_esp+0x18/0x71 (52)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: print_traces+0x17/0x50 / (0x0)

BUG: scheduling while atomic: bash/0x00000001/24154
caller is down_write+0x14d/0x17c
 [<c039d8e3>] __sched_text_start+0xbef/0xc3f (8)
 [<c039e946>] down_write+0x14d/0x17c (8)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (16)
 [<c039e946>] down_write+0x14d/0x17c (52)
 [<c026d262>] generic_unplug_device+0x32/0x50 (40)
 [<c026d29b>] blk_backing_dev_unplug+0x1b/0x1d (16)
 [<c01558bc>] swap_unplug_io_fn+0x58/0xb5 (8)
 [<c0162583>] block_sync_page+0x4f/0x58 (36)
 [<c013e5c4>] sync_page+0x50/0x59 (12)
 [<c039e5c0>] __wait_on_bit_lock+0x53/0x61 (8)
 [<c013e574>] sync_page+0x0/0x59 (8)
 [<c013ed4b>] __lock_page+0xa3/0xab (20)
 [<c01335a7>] wake_bit_function+0x0/0x55 (24)
 [<c01335a7>] wake_bit_function+0x0/0x55 (32)
 [<c014e9ce>] do_swap_page+0x219/0x2da (20)
 [<c014f175>] handle_mm_fault+0xdf/0x18e (52)
 [<c0117355>] do_page_fault+0x1be/0x595 (48)
 [<c039f019>] _spin_unlock+0x12/0x2d (60)
 [<c015d40e>] vfs_read+0xbf/0x12a (88)
 [<c015d6df>] sys_read+0x51/0x80 (44)
 [<c0117197>] do_page_fault+0x0/0x595 (12)
 [<c0105d3d>] error_code+0x2d/0x38 (8)
 [<c010521f>] sysenter_past_esp+0x18/0x71 (52)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: print_traces+0x17/0x50 / (0x0)

BUG: scheduling while atomic: bash/0x00000001/24154
caller is io_schedule+0x26/0x30
 [<c039d8e3>] __sched_text_start+0xbef/0xc3f (8)
 [<c039e2c8>] io_schedule+0x26/0x30 (8)
 [<c039f019>] _spin_unlock+0x12/0x2d (28)
 [<c039f019>] _spin_unlock+0x12/0x2d (52)
 [<c039e2c8>] io_schedule+0x26/0x30 (36)
 [<c013e5b8>] sync_page+0x44/0x59 (12)
 [<c039e5c0>] __wait_on_bit_lock+0x53/0x61 (8)
 [<c013e574>] sync_page+0x0/0x59 (8)
 [<c013ed4b>] __lock_page+0xa3/0xab (20)
 [<c01335a7>] wake_bit_function+0x0/0x55 (24)
 [<c01335a7>] wake_bit_function+0x0/0x55 (32)
 [<c014e9ce>] do_swap_page+0x219/0x2da (20)
 [<c014f175>] handle_mm_fault+0xdf/0x18e (52)
 [<c0117355>] do_page_fault+0x1be/0x595 (48)
 [<c039f019>] _spin_unlock+0x12/0x2d (60)
 [<c015d40e>] vfs_read+0xbf/0x12a (88)
 [<c015d6df>] sys_read+0x51/0x80 (44)
 [<c0117197>] do_page_fault+0x0/0x595 (12)
 [<c0105d3d>] error_code+0x2d/0x38 (8)
 [<c010521f>] sysenter_past_esp+0x18/0x71 (52)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: print_traces+0x17/0x50 / (0x0)

BUG: sleeping function called from invalid context bash(24154) at kernel/mutex.c:25
in_atomic():1 [00000001], irqs_disabled():0
 [<c011c4c7>] __might_sleep+0xbb/0xc9 (8)
 [<c0133970>] _mutex_lock+0x1f/0x3b (36)
 [<c014e847>] do_swap_page+0x92/0x2da (16)
 [<c014f175>] handle_mm_fault+0xdf/0x18e (52)
 [<c0117355>] do_page_fault+0x1be/0x595 (48)
 [<c039f019>] _spin_unlock+0x12/0x2d (60)
 [<c015d40e>] vfs_read+0xbf/0x12a (88)
 [<c015d6df>] sys_read+0x51/0x80 (44)
 [<c0117197>] do_page_fault+0x0/0x595 (12)
 [<c0105d3d>] error_code+0x2d/0x38 (8)
 [<c010521f>] sysenter_past_esp+0x18/0x71 (52)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: print_traces+0x17/0x50 / (0x0)

BUG: scheduling while atomic: bash/0x04000001/24154
caller is cond_resched+0x5e/0x7f
 [<c039d8e3>] __sched_text_start+0xbef/0xc3f (8)
 [<c039e264>] cond_resched+0x5e/0x7f (8)
 [<c011f16e>] vprintk+0x11e/0x15b (24)
 [<c0133f74>] check_preempt_timing+0x70/0x1aa (16)
 [<c0106116>] dump_stack+0x1c/0x20 (40)
 [<c039e264>] cond_resched+0x5e/0x7f (36)
 [<c0133975>] _mutex_lock+0x24/0x3b (16)
 [<c014e847>] do_swap_page+0x92/0x2da (16)
 [<c014f175>] handle_mm_fault+0xdf/0x18e (52)
 [<c0117355>] do_page_fault+0x1be/0x595 (48)
 [<c039f019>] _spin_unlock+0x12/0x2d (60)
 [<c015d40e>] vfs_read+0xbf/0x12a (88)
 [<c015d6df>] sys_read+0x51/0x80 (44)
 [<c0117197>] do_page_fault+0x0/0x595 (12)
 [<c0105d3d>] error_code+0x2d/0x38 (8)
 [<c010521f>] sysenter_past_esp+0x18/0x71 (52)
preempt count: 04000002
. 2-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: print_traces+0x17/0x50 / (0x0)

BUG: scheduling while atomic: bash/0x00000001/24154
caller is do_exit+0x2ee/0x3b1
 [<c039d8e3>] __sched_text_start+0xbef/0xc3f (8)
 [<c01219a6>] do_exit+0x2ee/0x3b1 (8)
 [<c01210a2>] exit_notify+0x280/0x896 (48)
 [<c01219a6>] do_exit+0x2ee/0x3b1 (68)
 [<c0121b82>] do_group_exit+0x91/0xb5 (36)
 [<c012acfd>] get_signal_to_deliver+0x1bf/0x396 (24)
 [<c01050ea>] do_signal+0xe3/0x11b (48)
 [<c039f019>] _spin_unlock+0x12/0x2d (40)
 [<c015d40e>] vfs_read+0xbf/0x12a (88)
 [<c015d6df>] sys_read+0x51/0x80 (44)
 [<c010517a>] do_notify_resume+0x58/0x5a (28)
 [<c01052f7>] work_notifysig+0x13/0x18 (12)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
.. entry 2: print_traces+0x17/0x50 / (0x0)

INIT: Switching to runlevel: 6
INIT: Sending processes the TERM signal
Stopping atd: [  OK  ]
Stopping keytable:  [  OK  ]
Stopping cups: [  OK  ]
Shutting down xfs: [  OK  ]
Shutting down console mouse services: [  OK  ]
Stopping sshd:[  OK  ]
Shutting down sendmail: [  OK  ]
Shutting down sm-client: [  OK  ]
Stopping xinetd: [  OK  ]
Stopping crond: [  OK  ]
Saving random seed:  [  OK  ]
Stopping NFS statd: 

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Adam Heath]

On Wed, 20 Oct 2004, Alexander Batyrshin wrote:

>
>
> Ingo Molnar wrote:
> > i have released the -U8 Real-Time Preemption patch:
> >
> >   http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U8
> >
>
> 2.
> if execute
> ``for i in `seq 1 9999`; do nohup bash >/dev/null 2>&1 & done'',
> then you'll get something like:
> [...skip...]
> Warning: dev (pts0) tty->count(16) != #fd's(8) in tty_open
> Warning: dev (pts0) tty->count(16) != #fd's(11) in tty_open
> Warning: dev (pts0) tty->count(17) != #fd's(13) in tty_open
> Warning: dev (pts0) tty->count(18) != #fd's(16) in tty_open
> Warning: dev (pts0) tty->count(18) != #fd's(16) in tty_open
> Warning: dev (pts0) tty->count(18) != #fd's(16) in tty_open
> Warning: dev (pts0) tty->count(19) != #fd's(16) in tty_open
> Warning: dev (pts0) tty->count(19) != #fd's(16) in release_dev
> Warning: dev (pts0) tty->count(18) != #fd's(16) in tty_open
> Warning: dev (pts0) tty->count(18) != #fd's(16) in tty_open
> Warning: dev (pts0) tty->count(19) != #fd's(17) in tty_open
> Warning: dev (pts0) tty->count(19) != #fd's(17) in release_dev
> Warning: dev (pts0) tty->count(18) != #fd's(17) in tty_open
> Warning: dev (pts0) tty->count(18) != #fd's(17) in tty_open
> Warning: dev (pts0) tty->count(19) != #fd's(17) in release_dev
> Warning: dev (pts0) tty->count(17) != #fd's(18) in release_dev
> Warning: dev (pts0) tty->count(16) != #fd's(19) in release_dev
> Warning: dev (pts0) tty->count(15) != #fd's(18) in release_dev
> Warning: dev (pts0) tty->count(14) != #fd's(18) in release_dev
> Warning: dev (pts0) tty->count(14) != #fd's(18) in release_dev
> Warning: dev (pts0) tty->count(13) != #fd's(18) in tty_open
> Warning: dev (pts0) tty->count(14) != #fd's(19) in release_dev
> Warning: dev (pts0) tty->count(13) != #fd's(18) in release_dev
> Warning: dev (pts0) tty->count(13) != #fd's(17) in release_dev
> Warning: dev (pts0) tty->count(12) != #fd's(18) in tty_open
> Warning: dev (pts0) tty->count(12) != #fd's(18) in release_dev
> Warning: dev (pts0) tty->count(12) != #fd's(18) in release_dev
> Warning: dev (pts0) tty->count(28) != #fd's(16) in tty_open
> Warning: dev (pts0) tty->count(528) != #fd's(13) in tty_open
> Warning: dev (pts0) tty->count(528) != #fd's(13) in release_dev
> Warning: dev (pts0) tty->count(527) != #fd's(12) in tty_open
> Warning: dev (pts0) tty->count(528) != #fd's(12) in release_dev
> Warning: dev (pts0) tty->count(538) != #fd's(28) in release_dev
> Warning: dev (pts0) tty->count(537) != #fd's(528) in tty_open
> Warning: dev (pts0) tty->count(537) != #fd's(527) in release_dev
> Warning: dev (pts0) tty->count(536) != #fd's(527) in tty_open
> Warning: dev (pts0) tty->count(536) != #fd's(527) in tty_open
> Warning: dev (pts0) tty->count(536) != #fd's(527) in release_dev
> Warning: dev (pts0) tty->count(535) != #fd's(527) in tty_open
> Warning: dev (pts0) tty->count(535) != #fd's(530) in tty_open
> [...skip...]
> Warning: dev (pts0) tty->count(11) != #fd's(71) in release_dev
> Warning: dev (pts0) tty->count(10) != #fd's(71) in release_dev
> Warning: dev (pts0) tty->count(9) != #fd's(71) in release_dev
> Warning: dev (pts0) tty->count(8) != #fd's(71) in release_dev
> Warning: dev (pts0) tty->count(7) != #fd's(71) in release_dev
> Warning: dev (pts0) tty->count(6) != #fd's(71) in release_dev
> Warning: dev (pts0) tty->count(5) != #fd's(71) in release_dev
> Warning: dev (pts0) tty->count(4) != #fd's(71) in release_dev
> Warning: dev (pts0) tty->count(3) != #fd's(71) in release_dev
> eggcups: page allocation failure. order:1, mode:0x20
>   [<c01439d8>] __alloc_pages+0x228/0x3ff (8)
>   [<c0143bce>] __get_free_pages+0x1f/0x3b (68)
>   [<c014719c>] kmem_getpages+0x25/0xe0 (8)
> [...skip...]

I got something like this too, just now.  Not doing anything special(tty7
holds xdm).

Warning: dev (pts7) tty->count(4) != #fd's(3) in tty_open
Warning: dev (pts7) tty->count(4) != #fd's(3) in release_dev

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Daniel Walker]

On Wed, 2004-10-20 at 10:49, Alexander Batyrshin wrote:
> Ingo Molnar wrote:
> > i have released the -U8 Real-Time Preemption patch:
> > 
> >   http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U8
> > 
> 
> used i386/defconfig
> 
> 1. at boot
> [...skip...]
> hda: 78242976 sectors (40060 MB) w/2048KiB Cache, CHS=16383/255/63, 
> UDMA(100)
>   hda: hda1 hda2 hda3 hda4 < hda5 >
> BUG: semaphore recursion deadlock detected!
> .. current task khpsbpkt/723 is already holding c04610c0.
> 00001f23 00001f2f c04e8de9 00000086 00000009 00000000 c011b552 00000020
>         00000400 c03b3efa dfdc9f70 dfdc9f50 0000000c dfdc78f0 c011b430 
> c03b3efa
>         dfdc9f70 c01052a5 c03b3efa c03b3efa 00000000 dfdc78f0 dfdc78f0 
> c01fd364
> Call Trace:
>   [<c012caa4>] __kernel_text_address+0x2e/0x37 (24)
>   [<c01051c9>] show_trace+0x4e/0xcc (12)
>   [<c01052c9>] show_stack+0x82/0x97 (36)
>   [<c01fd364>] __rwsem_deadlock+0xd9/0x135 (24)
>   [<c039e2a0>] down_write_interruptible+0xe6/0x202 (48)
>   [<c029dd80>] hpsbpkt_thread+0x2b/0x86 (48)
>   [<c029dd55>] hpsbpkt_thread+0x0/0x86 (12)
>   [<c01024d1>] kernel_thread_helper+0x5/0xb (4)
> preempt count: 00000003
> . 3-level deep critical section nesting:
> .. entry 1: _spin_lock_irqsave+0x19/0x74 / (0x0)
> .. entry 2: _spin_lock+0x19/0x6d / (0x0)
> .. entry 3: print_traces+0x17/0x50 / (0x0)
> [...skip...]


This looks related to IEEE1394 . It has a deadlock in it. Try turning it
off..



Daniel

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

* Alexander Batyrshin <abatyrshin@ru.mvista.com> wrote:

> used i386/defconfig

> BUG: semaphore recursion deadlock detected!
> .. current task khpsbpkt/723 is already holding c04610c0.

ok, this should be fixed in -U9.2.

> 2.
> if execute
> ``for i in `seq 1 9999`; do nohup bash >/dev/null 2>&1 & done'',
> then you'll get something like:
> [...skip...]
> Warning: dev (pts0) tty->count(16) != #fd's(8) in tty_open
> Warning: dev (pts0) tty->count(16) != #fd's(11) in tty_open

> I'v tested it against linux-2.6.9-rc4-mm1 => all was ok

i have trouble reproducing this myself. Can you still trigger it under
-U9.2? If yes then could you check whether this still happens with a the
same .config but with CONFIG_SMP turned off? This smells like a locking
bug/breakage in the tty layer that we dont detect. You have all the
relevant debug options turned on, correct? (DEBUG_PREEMPT and
RWSEM_DEADLOCK_DETECT)

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

* Alexander Batyrshin <abatyrshin@ru.mvista.com> wrote:

> 2.
> if execute
> ``for i in `seq 1 9999`; do nohup bash >/dev/null 2>&1 & done'',
> then you'll get something like:
> [...skip...]
> Warning: dev (pts0) tty->count(16) != #fd's(8) in tty_open
> Warning: dev (pts0) tty->count(16) != #fd's(11) in tty_open

btw., where did you run the test - over ssh or in an xterm?

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Esben Nielsen]

Hi,
 I finally got around to get my labtop up running with this. It works -
nearly that is - X even started up!

When I start the network I get the trace below. And my network runs
awfully slowly. Could seem like the interrupt doesn't work correctly....

Regards,
Esben

Oct 20 21:45:53 localhost kernel: e100: Intel(R) PRO/100 Network Driver,
3.1.4-k
2-NAPI
Oct 20 21:45:53 localhost kernel: e100: Copyright(c) 1999-2004 Intel
Corporation
Oct 20 21:45:53 localhost kernel: ACPI: PCI interrupt 0000:00:09.0[A] ->
GSI 11 
(level, low) -> IRQ 11
Oct 20 21:45:53 localhost kernel: e100: eth0: e100_probe: addr 0x41280000,
irq 1
1, MAC addr 00:D0:59:2D:91:84
Oct 20 21:45:53 localhost kernel: ip/1988: BUG in enable_irq at
/misc/frodo_opt3
/simlo/Linux-RT/linux-2.6.9-rc4-mm1-rt-u8.1/kernel/irq/manage.c:111
Oct 20 21:45:53 localhost kernel:  [<c013614e>] enable_irq+0xee/0x100 (12)
Oct 20 21:45:53 localhost kernel:  [<d089741e>] e100_up+0x10e/0x200 [e100]
(48)
Oct 20 21:45:54 localhost kernel:  [<d08987c0>] e100_open+0x30/0x80 [e100]
(48)
Oct 20 21:45:54 localhost kernel:  [<c010fe00>] mcount+0x14/0x18 (12)
Oct 20 21:45:54 localhost kernel:  [<c02ea108>] dev_open+0x88/0xa0 (20)
Oct 20 21:45:54 localhost kernel:  [<c02eb82d>]
dev_change_flags+0x5d/0x140 (28)
Oct 20 21:45:54 localhost kernel:  [<c02e976e>] __dev_get_by_name+0xe/0xd0
(8)
Oct 20 21:45:54 localhost kernel:  [<c03293b7>] devinet_ioctl+0x277/0x6e0
(28)
Oct 20 21:45:54 localhost kernel:  [<c032b834>] inet_ioctl+0x64/0xb0 (108)
Oct 20 21:45:54 localhost kernel:  [<c02e0ae8>] sock_ioctl+0xc8/0x250 (28)
Oct 20 21:45:54 localhost kernel:  [<c016a319>] sys_ioctl+0xc9/0x230 (32)
Oct 20 21:45:54 localhost kernel:  [<c01046fd>]
sysenter_past_esp+0x52/0x71 (44)
Oct 20 21:45:54 localhost kernel: preempt count: 00000002
Oct 20 21:45:54 localhost kernel: . 2-level deep critical section nesting:
Oct 20 21:45:54 localhost kernel: .. entry 1: enable_irq+0x2e/0x100 /
(e100_up+0
x10e/0x200 [e100])
Oct 20 21:45:54 localhost kernel: .. entry 2: print_traces+0x1d/0x60 /
(dump_sta
ck+0x23/0x30)
Oct 20 21:45:54 localhost kernel: 
Oct 20 21:45:54 localhost kernel: e100: eth0: e100_watchdog: link up, 
10Mbps, ha



On Wed, 20 Oct 2004, Ingo Molnar wrote:

> 
> i have released the -U8 Real-Time Preemption patch:
> 
>   http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U8
> 
> this too is a fixes-only release. It includes the many semaphore-abuse
> and sleep_on() fixes/improvements from Thomas Gleixner, and it also
> includes a couple of semaphore related fixes.
> 
> I believe the semaphore fixes should resolve a number of the deadlocks
> reported for -U7.
> 
> In particular it seems the only sane and reliable way to convert RCU
> locking was to allow the following semantics for rwsems: allow reads to
> nest, and allow self-read-recursion of a self-write-held semaphore. My
> current implementation for this allows semaphore unfairness, but that
> can be fixed later on. Most importantly, the RCU to RT-locking
> conversions are much more automatic now and map nicely to what the code
> is doing upstream. Most of the time they involve a conversion of a
> spinlock or semaphore into a rwlock or rwsem. The old code maps to new
> code almost automatically, the only manual work needed was to associate
> the rcu_read_lock() with the writers-lock that it excludes against,
> which is a pretty clear (but not automatic, and hence not automatable)
> decision. This way i could convert some more networking code, and
> simplify the older changes and hopefully get rid of some deadlocks. The
> locking API is still not in its final form, but it's getting closer.
> 
> Changes since -U7:
> 
>  - deadlock fix: sysfs/driver-base semaphore fixes from Thomas Gleixner
> 
>  - deadlock fix: scsi semaphore fixes from Thomas Gleixner
> 
>  - NFS sleep_on() fixes from Thomas Gleixner
> 
>  - rawmidid.c sleep_on() fix from Thomas Gleixner
> 
>  - [ i've added more wait_for_completion_*() primitives, to ease 
>      conversion of other semaphore-(ab-)using code. ]
> 
>  - make rwsems self-recursive
> 
>  - RCU lock conversion: convert rtnl_sem RCU use.
> 
>  - netfilter deadlock fix - clean up RCU locking.
> 
> to create a -U8 tree from scratch, the patching order is:
> 
>    http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.8.tar.bz2
>  + http://kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.9-rc4.bz2
>  + http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.9-rc4/2.6.9-rc4-mm1/2.6.9-rc4-mm1.bz2
>  + http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U8
> 
> 	Ingo
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Fernando Pablo Lopez-Lezcano]

On Wed, 2004-10-20 at 02:45, Ingo Molnar wrote:
> i have released the -U8 Real-Time Preemption patch:
> 
>   http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U8
> 
> this too is a fixes-only release.

A bit late, but a short report. I managed to boot into U8.1 (Athlon64 up
machine). But only if I turn off kudzu (the hardware discovery program
in FC2). If I leave it on, I get tons of kernel error or warning
messages on the console, they scroll too fast for me to read, once that
starts the machine is pretty much stuck - the messages keep scrolling
but that's about it. I have not managed to capture the output to see
what they say...

If I boot without kudzu I can use sound, but I have not done yet any
latency testing. 

-- Fernando

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Rui Nuno Capela]

[-- Attachment #1: Type: text/plain, Size: 14915 bytes --]

Ingo Molnar wrote:
>
> i have released the -U8 Real-Time Preemption patch:
>
>   http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U8
>

Hi,

   I'm posting this time about to report my current status of my two
boxes, regarding the realtime-preempt-2.6.9-rc4-mm1-U8 patch. So it
seems that now the positions have been somewhat reversed. Respective
config.gz are attached.


a) Desktop P4 2.80C@3.3Ghz SMP/HT (SuSE 9.1 Pro)
   config-2.6.9-rc4-mm1-RT-U8.0smp.gz

   This is apparently but surprinsingly OK, as everything seems to work
flawlessly, besides some quirks on the onboard NIC (sk98lin) that only
shows up initially but stabilizes later on. Indeed, U8 is the first
SMP+PREEMPT_REALTIME encarnation that runs at all and is fairly
workable on this machine. This is a relief.


b) Laptop P4 2.533Ghz UP (Mandrake 10.1c)
   config-2.6.9-rc4-mm1-RT-U8.1.gz

   This box was known to work without major issues until U4. With U8 it's
a real pain. Once trivial operations turns out fatal now. Running jackd
-R, which has been a flagship before, now freezes the whole system in
no time. (I'll take some netconsole capture sessions later)

   One of the signs that there's real trouble in here can be seen on the
following complete dmesg output (which was even a miracle to be
captured at all). This shows the complete bootstrap and init sequences
and at the end one fatal crash while plugging an USB flash memory stick
(usb-storage). This has been already reported earlier yesterday, but I
just want to make it here, as the evidence-at-hand.

   After this precise occurence, the system becomes very flaky, unreliable
and often ends up freezing to death.


Hope this helps (me)

Bye now.
--
Linux version 2.6.9-rc4-mm1-RT-U8.1 (root@lambda) (gcc version 3.4.1
(Mandrakelinux (Alpha 3.4.1-3mdk)) #1 Thu Oct 21 09:08:18 WEST 2004
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009f800 (usable)
 BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000dc000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000001f770000 (usable)
 BIOS-e820: 000000001f770000 - 000000001f77f000 (ACPI data)
 BIOS-e820: 000000001f77f000 - 000000001f780000 (ACPI NVS)
 BIOS-e820: 000000001f780000 - 000000001f800000 (reserved)
 BIOS-e820: 000000002f780000 - 000000002f800000 (reserved)
 BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
503MB LOWMEM available.
On node 0 totalpages: 128880
  DMA zone: 4096 pages, LIFO batch:1
  Normal zone: 124784 pages, LIFO batch:16
  HighMem zone: 0 pages, LIFO batch:1
DMI 2.3 present.
ACPI: RSDP (v000 PTLTD                                 ) @ 0x000f6c70
ACPI: RSDT (v001 PTLTD    RSDT   0x06040000  LTP 0x00000000) @ 0x1f7783fd
ACPI: FADT (v001 ATI    Salmon   0x06040000 ATI  0x000f4240) @ 0x1f77ef64
ACPI: BOOT (v001 PTLTD  $SBFTBL$ 0x06040000  LTP 0x00000001) @ 0x1f77efd8
ACPI: DSDT (v001    ATI MS2_1535 0x06040000 MSFT 0x0100000e) @ 0x00000000
Built 1 zonelists
No local APIC present or hardware disabled
Initializing CPU#0
Kernel command line: BOOT_IMAGE=linux-RT-U8.1 ro root=305 devfs=nomount
acpi=on
PID hash table entries: 2048 (order: 11, 32768 bytes)
Detected 2525.698 MHz processor.
Using tsc for high-res timesource
Console: colour VGA+ 80x25
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
Memory: 507728k/515520k available (1791k kernel code, 7300k reserved, 591k
data, 152k init, 0k highmem)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Calibrating delay loop... 4931.58 BogoMIPS (lpj=2465792)
Security Scaffold v1.0.0 initialized
Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
CPU: After generic identify, caps: bfebf9ff 00000000 00000000 00000000
CPU: After vendor identify, caps:  bfebf9ff 00000000 00000000 00000000
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: After all inits, caps:        bfebf9ff 00000000 00000000 00000080
CPU: Intel(R) Pentium(R) 4 CPU 2.53GHz stepping 07
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... OK.
ACPI: IRQ9 SCI: Edge set to Level Trigger.
ksoftirqd started up.
NET: Registered protocol family 16
PCI: PCI BIOS revision 2.10 entry at 0xfd88b, last bus=2
PCI: Using configuration type 1
mtrr: v2.0 (20020519)
ACPI: Subsystem revision 20040816
ACPI: Interpreter enabled
ACPI: Using PIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (00:00)
PCI: Probing PCI hardware (bus 00)
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.AGPB._PRT]
ACPI: PCI Interrupt Link [LNK0] (IRQs 7 10) *0, disabled.
ACPI: PCI Interrupt Link [LNK1] (IRQs 7 *10)
ACPI: PCI Interrupt Link [LNK2] (IRQs 7 10) *0, disabled.
ACPI: PCI Interrupt Link [LNK3] (IRQs 7 10) *0, disabled.
ACPI: PCI Interrupt Link [LNK4] (IRQs 7 *10)
ACPI: PCI Interrupt Link [LNK5] (IRQs 7 *11)
ACPI: PCI Interrupt Link [LNK6] (IRQs 7 10) *0, disabled.
ACPI: PCI Interrupt Link [LNK7] (IRQs *5)
ACPI: PCI Interrupt Link [LNK8] (IRQs 7 *10)
ACPI: Embedded Controller [EC0] (gpe 24)
SCSI subsystem initialized
PCI: Using ACPI for IRQ routing
** PCI interrupts are no longer routed automatically.  If this
** causes a device to stop working, it is probably because the
** driver failed to call pci_enable_device().  As a temporary
** workaround, the "pci=routeirq" argument restores the old
** behavior.  If this argument makes the device work again,
** please email the output of "lspci" to bjorn.helgaas@hp.com
** so I can fix the driver.
Simple Boot Flag at 0x37 set to 0x1
devfs: 2004-01-31 Richard Gooch (rgooch@atnf.csiro.au)
devfs: boot_options: 0x0
Activating ISA DMA hang workarounds.
Real Time Clock Driver v1.12
ACPI: PS/2 Keyboard Controller [KBC0] at I/O 0x60, 0x64, irq 1
ACPI: PS/2 Mouse Controller [MSE0] at irq 12
serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
Serial: 8250/16550 driver $Revision: 1.90 $ 8 ports, IRQ sharing disabled
ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
PCI: Enabling device 0000:00:08.0 (0000 -> 0003)
ACPI: PCI Interrupt Link [LNK6] enabled at IRQ 10
PCI: setting IRQ 10 as level-triggered
ACPI: PCI interrupt 0000:00:08.0[A] -> GSI 10 (level, low) -> IRQ 10
ttyS0 at I/O 0x1428 (irq = 10) is a 8250
ttyS2 at I/O 0x1440 (irq = 10) is a 8250
ttyS3 at I/O 0x1450 (irq = 10) is a 8250
ttyS4 at I/O 0x1460 (irq = 10) is a 8250
ttyS5 at I/O 0x1470 (irq = 10) is a 8250
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
ACPI: Floppy Controller [FDC] at I/O 0x3f0-0x3f5, 0x3f7 irq 6 dma channel 2
elevator: using anticipatory as default io scheduler
Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
ALI15X3: IDE controller at PCI slot 0000:00:10.0
ACPI: PCI interrupt 0000:00:10.0[A]: no GSI
ALI15X3: chipset revision 196
ALI15X3: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0x2000-0x2007, BIOS settings: hda:DMA, hdb:pio
    ide1: BM-DMA at 0x2008-0x200f, BIOS settings: hdc:pio, hdd:pio
Probing IDE interface ide0...
hda: IC25N040ATCS04-0, ATA DISK drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Probing IDE interface ide1...
hdc: HL-DT-STCD-RW/DVD DRIVE GCC-4240N, ATAPI CD/DVD-ROM drive
ide1 at 0x170-0x177,0x376 on irq 15
Probing IDE interface ide2...
ide2: Wait for ready failed before probe !
Probing IDE interface ide3...
ide3: Wait for ready failed before probe !
Probing IDE interface ide4...
ide4: Wait for ready failed before probe !
Probing IDE interface ide5...
ide5: Wait for ready failed before probe !
hda: max request size: 128KiB
hda: 78140160 sectors (40007 MB) w/1768KiB Cache, CHS=65535/16/63, UDMA(100)
hda: cache flushes not supported
 /dev/ide/host0/bus0/target0/lun0: p1 p2 < p5 p6 p7 >
hdc: ATAPI 24X DVD-ROM CD-R/RW drive, 2048kB Cache, DMA
Uniform CD-ROM driver Revision: 3.20
mice: PS/2 mouse device common for all mice
input: AT Translated Set 2 keyboard on isa0060/serio0
Synaptics Touchpad, model: 1
 Firmware: 5.8
 Sensor: 35
 new absolute packet format
 Touchpad has extended capability bits
 -> multifinger detection
 -> palm detection
input: SynPS/2 Synaptics TouchPad on isa0060/serio1
NET: Registered protocol family 2
IP: routing cache hash table of 128 buckets, 20Kbytes
TCP: Hash tables configured (established 1024 bind 1638)
NET: Registered protocol family 1
NET: Registered protocol family 17
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 152k freed
kjournald starting.  Commit interval 5 seconds
ACPI: AC Adapter [ACAD] (on-line)
ACPI: Battery Slot [BAT1] (battery present)
ACPI: Power Button (FF) [PWRF]
ACPI: Lid Switch [LID]
ACPI: Processor [CPU0] (supports C1 C2)
ACPI: Thermal Zone [THRM] (52 C)
usbcore: registered new driver usbfs
usbcore: registered new driver hub
ohci_hcd: 2004 Feb 02 USB 1.1 'Open' Host Controller (OHCI) Driver (PCI)
ACPI: PCI Interrupt Link [LNK8] enabled at IRQ 10
ACPI: PCI interrupt 0000:00:02.0[A] -> GSI 10 (level, low) -> IRQ 10
ohci_hcd 0000:00:02.0: OHCI Host Controller
ohci_hcd 0000:00:02.0: irq 10, pci mem 0xd4000000
ohci_hcd 0000:00:02.0: new USB bus registered, assigned bus number 1
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 3 ports detected
ACPI: PCI Interrupt Link [LNK4] enabled at IRQ 10
ACPI: PCI interrupt 0000:00:0f.0[A] -> GSI 10 (level, low) -> IRQ 10
ohci_hcd 0000:00:0f.0: OHCI Host Controller
ohci_hcd 0000:00:0f.0: irq 10, pci mem 0xd4009000
ohci_hcd 0000:00:0f.0: new USB bus registered, assigned bus number 2
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 3 ports detected
usb usb2: string descriptor 0 read error: -113
usb usb2: string descriptor 0 read error: -113
usb usb2: string descriptor 0 read error: -113
usb usb1: string descriptor 0 read error: -113
usb usb1: string descriptor 0 read error: -113
usb usb1: string descriptor 0 read error: -113
usb usb2: string descriptor 0 read error: -113
usb usb2: string descriptor 0 read error: -113
usb usb2: string descriptor 0 read error: -113
usb usb1: string descriptor 0 read error: -113
usb usb1: string descriptor 0 read error: -113
usb usb1: string descriptor 0 read error: -113
EXT3 FS on hda5, internal journal
Adding 506008k swap on /dev/hda6.  Priority:-1 extents:1
kjournald starting.  Commit interval 5 seconds
EXT3 FS on hda7, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
subfs 0.9
loop: loaded (max 8 devices)
natsemi dp8381x driver, version 1.07+LK1.0.17, Sep 27, 2002
  originally by Donald Becker <becker@scyld.com>
  http://www.scyld.com/network/natsemi.html
  2.4.x kernel port by Jeff Garzik, Tjeerd Mulder
ACPI: PCI Interrupt Link [LNK1] enabled at IRQ 10
ACPI: PCI interrupt 0000:00:12.0[A] -> GSI 10 (level, low) -> IRQ 10
natsemi eth0: NatSemi DP8381[56] at 0xd400a000 (0000:00:12.0),
00:0b:cd:85:0f:54, IRQ 10, port TP.
Linux Kernel Card Services
  options:  [pci] [cardbus] [pm]
PCI: Enabling device 0000:00:0a.0 (0005 -> 0007)
ACPI: PCI Interrupt Link [LNK5] enabled at IRQ 11
PCI: setting IRQ 11 as level-triggered
ACPI: PCI interrupt 0000:00:0a.0[A] -> GSI 11 (level, low) -> IRQ 11
Yenta: CardBus bridge found at 0000:00:0a.0 [103c:0850]
Yenta: ISA IRQ mask 0x00b8, PCI irq 11
Socket status: 30000007
cs: IO port probe 0x0c00-0x0cff: clean.
cs: IO port probe 0x0100-0x04ff: excluding 0x408-0x40f 0x480-0x48f
0x4d0-0x4d7
cs: IO port probe 0x0a00-0x0aff: clean.
prism2_cs: Ignoring new-style parameters in presence of obsolete ones
prism2cs_init: prism2_cs.o: 0.2.1-pre22 Loaded
prism2cs_init: dev_info is: prism2_cs
PCI: Enabling device 0000:00:06.0 (0005 -> 0007)
ACPI: PCI Interrupt Link [LNK7] enabled at IRQ 5
PCI: setting IRQ 5 as level-triggered
ACPI: PCI interrupt 0000:00:06.0[A] -> GSI 5 (level, low) -> IRQ 5
usbcore: registered new driver snd-usb-usx2y
Realtime LSM initialized (group 81, mlock=1)
mtrr: no more MTRRs available
ohci_hcd 0000:00:0f.0: wakeup
usb 2-1: new full speed USB device using address 2
Initializing USB Mass Storage driver...
scsi0 : SCSI emulation for USB Mass Storage devices
usbcore: registered new driver usb-storage
USB Mass Storage support registered.
usb-storage: device found at 2
usb-storage: waiting for device to settle before scanning
------------[ cut here ]------------
kernel BUG at lib/rwsem-generic.c:598!
invalid operand: 0000 [#1]
PREEMPT
Modules linked in: usb_storage realtime commoncap snd_seq_oss
snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss snd_usb_usx2y
snd_usb_lib snd_rawmidi snd_seq_device snd_hwdep snd_ali5451
snd_ac97_codec snd_pcm snd_timer snd_page_alloc snd soundcore prism2_cs
p80211 ds yenta_socket pcmcia_core natsemi crc32 loop subfs evdev ohci_hcd
usbcore thermal processor fan button battery ac
CPU:    0
EIP:    0060:[<c01b7e24>]    Not tainted VLI
EFLAGS: 00010202   (2.6.9-rc4-mm1-RT-U8.1)
EIP is at up_write+0x1d4/0x202
eax: d4dce000   ebx: 00000292   ecx: d4e18980   edx: d52da030
esi: d4e69e24   edi: d5803400   ebp: d4dcfd6c   esp: d4dcfd4c
ds: 007b   es: 007b   ss: 0068   preempt: 00000001
Process usb-stor (pid: 6630, threadinfo=d4dce000 task=d4dcd8f0)
Stack: d4dcd8f0 d4dcfd78 c02bea7a 00000001 d4dcd8f0 00000292 d4e18980
d5803400
       d4dcfd84 e018e139 d4e18980 d4dcfd84 00000292 d58034b8 d4dcfdac
c022ed0c
       d4e18980 c022ef10 c023166d 00000000 d4e189d4 d4e18980 dcf21000
d5803400
Call Trace:
 [<c0104eb0>] show_stack+0x80/0x96 (28)
 [<c010504b>] show_registers+0x165/0x1de (56)
 [<c010525d>] die+0xf6/0x191 (64)
 [<c0105797>] do_invalid_op+0x10b/0x10d (188)
 [<c0104b0d>] error_code+0x2d/0x38 (100)
 [<e018e139>] queuecommand+0x70/0x7c [usb_storage] (24)
 [<c022ed0c>] scsi_dispatch_cmd+0x168/0x218 (40)
 [<c02342e1>] scsi_request_fn+0x1ee/0x42b (52)
 [<c0205606>] blk_insert_request+0xcd/0xfb (44)
 [<c0232f43>] scsi_insert_special_req+0x3b/0x3f (28)
 [<c0233175>] scsi_wait_req+0x61/0x94 (60)
 [<c0235290>] scsi_probe_lun+0x8e/0x240 (68)
 [<c0235883>] scsi_probe_and_add_lun+0xb0/0x1be (48)
 [<c0236009>] scsi_scan_target+0xa4/0x123 (60)
 [<c0236115>] scsi_scan_channel+0x8d/0xa4 (48)
 [<c02361a5>] scsi_scan_host_selected+0x79/0xd4 (44)
 [<c0236231>] scsi_scan_host+0x31/0x33 (28)
 [<e0190cbd>] usb_stor_scan_thread+0x144/0x155 [usb_storage] (96)
 [<c0102305>] kernel_thread_helper+0x5/0xb (723714068)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: die+0x3a/0x191 / (do_invalid_op+0x10b/0x10d)
.. entry 2: print_traces+0x16/0x4a / (show_stack+0x80/0x96)

Code: e8 af f9 ff ff 89 f8 e8 fd af f5 ff e9 35 ff ff ff 0f 0b a5 00 e3 e8
2c c0 e9 da fe ff ff 0f 0b a4 00 e3 e8 2c c0 e9 c4 fe ff ff <0f> 0b 56 02
6f 75 2d c0 e9 3c fe ff ff e8 a8 5b 10 00 e9 22 ff
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

[-- Attachment #2: config-2.6.9-rc4-mm1-RT-U8.0smp.gz --]
[-- Type: application/x-gzip-compressed, Size: 7590 bytes --]

[-- Attachment #3: config-2.6.9-rc4-mm1-RT-U8.1.gz --]
[-- Type: application/x-gzip-compressed, Size: 8276 bytes --]

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Thomas Gleixner]

On Thu, 2004-10-21 at 11:12, Rui Nuno Capela wrote:
>  [<e018e139>] queuecommand+0x70/0x7c [usb_storage] (24)

As I already pointed out, this is a problem due to up(sema) in
queuecommand. That's one of the semaphore abuse points, which needs to
be fixed. 

The problem is that semaphores are hold by Process A and released by
Process B, which makes Ingo's checks trigger

tglx

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Christoph Hellwig]

On Thu, Oct 21, 2004 at 11:16:30AM +0200, Thomas Gleixner wrote:
> On Thu, 2004-10-21 at 11:12, Rui Nuno Capela wrote:
> >  [<e018e139>] queuecommand+0x70/0x7c [usb_storage] (24)
> 
> As I already pointed out, this is a problem due to up(sema) in
> queuecommand. That's one of the semaphore abuse points, which needs to
> be fixed. 
> 
> The problem is that semaphores are hold by Process A and released by
> Process B, which makes Ingo's checks trigger

Which is perfectly valid for a semaphore.

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

* Christoph Hellwig <hch@infradead.org> wrote:

> > The problem is that semaphores are hold by Process A and released by
> > Process B, which makes Ingo's checks trigger
> 
> Which is perfectly valid for a semaphore.

yes, it is valid and perfectly fine code, but i'm trying to separate out
the simple 'mutex' functionality (99% of the semaphore users are just
that) and implement a 'counted semaphore' separately. This removes a
number of implementational constraints from mutexes.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Christoph Hellwig]

On Thu, Oct 21, 2004 at 11:44:38AM +0200, Ingo Molnar wrote:
> 
> * Christoph Hellwig <hch@infradead.org> wrote:
> 
> > > The problem is that semaphores are hold by Process A and released by
> > > Process B, which makes Ingo's checks trigger
> > 
> > Which is perfectly valid for a semaphore.
> 
> yes, it is valid and perfectly fine code, but i'm trying to separate out
> the simple 'mutex' functionality (99% of the semaphore users are just
> that) and implement a 'counted semaphore' separately. This removes a
> number of implementational constraints from mutexes.

So leave the good old struct semaphore alone and introduce a mutex_t..

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

* Christoph Hellwig <hch@infradead.org> wrote:

> > yes, it is valid and perfectly fine code, but i'm trying to separate out
> > the simple 'mutex' functionality (99% of the semaphore users are just
> > that) and implement a 'counted semaphore' separately. This removes a
> > number of implementational constraints from mutexes.
> 
> So leave the good old struct semaphore alone and introduce a mutex_t..

with nearly 1000 'struct semaphore' references in the kernel and 980 of
them being simple mutex use this is rather impractical. So i instead
went for safely detecting the 20 non-mutex uses and converting those
places. (Btw., 90% of those 20 cases can be detected safely at
compile-time (and link-time) by removing DECLARE_MUTEX_LOCKED and making
sema_init() a macro that only allows constant values of 0 and 1 and
produces a link error for other cases.)

this work is still incomplete so i'm not arguing for upstream inclusion.

(But while we did this a couple of places did turn out to use semaphores
for completion which is inefficient - we converted those to completions
and are contributing those changes to mainline. But this issue is
totally orthogonal to the issue of counted semaphores.)

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Thomas Gleixner]

On Thu, 2004-10-21 at 11:35, Christoph Hellwig wrote:
> On Thu, Oct 21, 2004 at 11:16:30AM +0200, Thomas Gleixner wrote:
> > On Thu, 2004-10-21 at 11:12, Rui Nuno Capela wrote:
> > >  [<e018e139>] queuecommand+0x70/0x7c [usb_storage] (24)
> > 
> > As I already pointed out, this is a problem due to up(sema) in
> > queuecommand. That's one of the semaphore abuse points, which needs to
> > be fixed. 
> > 
> > The problem is that semaphores are hold by Process A and released by
> > Process B, which makes Ingo's checks trigger
> 
> Which is perfectly valid for a semaphore.
> 

In fact this is used where wait_for_completion() is the correct thing to
do. It's not waiting for a resource. It's waiting for completion of a
commoand.

tglx

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Jens Axboe]

On Thu, Oct 21 2004, Thomas Gleixner wrote:
> On Thu, 2004-10-21 at 11:12, Rui Nuno Capela wrote:
> >  [<e018e139>] queuecommand+0x70/0x7c [usb_storage] (24)
> 
> As I already pointed out, this is a problem due to up(sema) in
> queuecommand. That's one of the semaphore abuse points, which needs to
> be fixed. 
> 
> The problem is that semaphores are hold by Process A and released by
> Process B, which makes Ingo's checks trigger

That's utter crap, it's perfectly valid use.

-- 
Jens Axboe

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Thomas Gleixner]

On Thu, 2004-10-21 at 11:53, Jens Axboe wrote:
> On Thu, Oct 21 2004, Thomas Gleixner wrote:
> > On Thu, 2004-10-21 at 11:12, Rui Nuno Capela wrote:
> > >  [<e018e139>] queuecommand+0x70/0x7c [usb_storage] (24)
> > 
> > As I already pointed out, this is a problem due to up(sema) in
> > queuecommand. That's one of the semaphore abuse points, which needs to
> > be fixed. 
> > 
> > The problem is that semaphores are hold by Process A and released by
> > Process B, which makes Ingo's checks trigger
> 
> That's utter crap, it's perfectly valid use.

It's not!

>From the code:

         init_MUTEX_LOCKED(&(us->sema));

This is used to wait for command completion and therefor we have the
completion API. It was used this way because the ancestor of completion
(sleep_on) was racy !

tglx

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Jens Axboe]

On Thu, Oct 21 2004, Thomas Gleixner wrote:
> On Thu, 2004-10-21 at 11:53, Jens Axboe wrote:
> > On Thu, Oct 21 2004, Thomas Gleixner wrote:
> > > On Thu, 2004-10-21 at 11:12, Rui Nuno Capela wrote:
> > > >  [<e018e139>] queuecommand+0x70/0x7c [usb_storage] (24)
> > > 
> > > As I already pointed out, this is a problem due to up(sema) in
> > > queuecommand. That's one of the semaphore abuse points, which needs to
> > > be fixed. 
> > > 
> > > The problem is that semaphores are hold by Process A and released by
> > > Process B, which makes Ingo's checks trigger
> > 
> > That's utter crap, it's perfectly valid use.
> 
> It's not!
> 
> >From the code:
> 
>          init_MUTEX_LOCKED(&(us->sema));
> 
> This is used to wait for command completion and therefor we have the
> completion API. It was used this way because the ancestor of completion
> (sleep_on) was racy !

I didn't look at the USB code, I'm just saying that it's perfectly valid
use of a semaphore the pattern you describe (process A holding it,
process B releasing it).

-- 
Jens Axboe

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Thomas Gleixner]

On Thu, 2004-10-21 at 12:11, Jens Axboe wrote:
> On Thu, Oct 21 2004, Thomas Gleixner wrote:
> > On Thu, 2004-10-21 at 11:53, Jens Axboe wrote:
> > > On Thu, Oct 21 2004, Thomas Gleixner wrote:
> > > > On Thu, 2004-10-21 at 11:12, Rui Nuno Capela wrote:
> > > > >  [<e018e139>] queuecommand+0x70/0x7c [usb_storage] (24)
> > > > 
> > > > As I already pointed out, this is a problem due to up(sema) in
> > > > queuecommand. That's one of the semaphore abuse points, which needs to
> > > > be fixed. 
> > > > 
> > > > The problem is that semaphores are hold by Process A and released by
> > > > Process B, which makes Ingo's checks trigger
> > > 
> > > That's utter crap, it's perfectly valid use.
> > 
> > It's not!
> > 
> > >From the code:
> > 
> >          init_MUTEX_LOCKED(&(us->sema));
> > 
> > This is used to wait for command completion and therefor we have the
> > completion API. It was used this way because the ancestor of completion
> > (sleep_on) was racy !
> 
> I didn't look at the USB code, I'm just saying that it's perfectly valid
> use of a semaphore the pattern you describe (process A holding it,
> process B releasing it).

Yeah, for a semaphore it is, but not for a mutex.

IMHO, this is not clearly seperated and therefor produces a lot of
confusion.

tglx

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

* Thomas Gleixner <tglx@linutronix.de> wrote:

> > > This is used to wait for command completion and therefor we have the
> > > completion API. It was used this way because the ancestor of completion
> > > (sleep_on) was racy !
> > 
> > I didn't look at the USB code, I'm just saying that it's perfectly valid
> > use of a semaphore the pattern you describe (process A holding it,
> > process B releasing it).
> 
> Yeah, for a semaphore it is, but not for a mutex.

but mutexes dont exist in upstream Linux as a separate entity. (they
exist in my tree but that's another ballgame.)

> IMHO, this is not clearly seperated and therefor produces a lot of
> confusion.

if used to complete some work then semaphores are indeed a tad unclean
and slightly slower than completions - but they are fully correct kernel
code. And there are much worse offenders of cleanliness around.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[john cooper]

Ingo Molnar wrote:

>* Thomas Gleixner <tglx@linutronix.de> wrote:
>
>
>>Yeah, for a semaphore it is, but not for a mutex.
>>
>
>but mutexes dont exist in upstream Linux as a separate entity. (they
>exist in my tree but that's another ballgame.)
>
Mutexes layered on existing semaphores seems convenient
at the moment. However a more native mutex mechanism
which tracks ownership would provide a basis for PI as
well as further instrumentation. This may not be an
issue at the present but I don't think it is too far
off.

-john




-- 
john.cooper@timesys.com

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Esben Nielsen]

On Thu, 21 Oct 2004, john cooper wrote:

> Ingo Molnar wrote:
> 
> >* Thomas Gleixner <tglx@linutronix.de> wrote:
> >
> >
> >>Yeah, for a semaphore it is, but not for a mutex.
> >>
> >
> >but mutexes dont exist in upstream Linux as a separate entity. (they
> >exist in my tree but that's another ballgame.)
> >
> Mutexes layered on existing semaphores seems convenient
> at the moment. However a more native mutex mechanism
> which tracks ownership would provide a basis for PI as
> well as further instrumentation. This may not be an
> issue at the present but I don't think it is too far
> off.
> 
> -john
> 

Actually you need to have another kind of semaphore based on a new kind of
wait-queue: Priority based. I.e. the task with the highest priority get
woken up first. Then on top of that you build your mutex.

I was planning to start to look at it and try to see if I could get
anything to work, but I must admit I haven't got much further than
just getting Igno's -U8.1 up running.

An idea I had was to make a macro in list.h called
 list_insert_sorted(list, element, condition_statement)
and use that in this kind of wait_queue...

To get a mutex with priority inheritance add an element pointing to the
current owner and a field where you store the owners original priority
which it has to be set back to when it relases the mutex (I am not sure
how this will work out if someone holds several mutexes!)

Regards,
Esben

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[john cooper]

Esben Nielsen wrote:

>On Thu, 21 Oct 2004, john cooper wrote:
>
>>Mutexes layered on existing semaphores seems convenient
>>at the moment. However a more native mutex mechanism
>>which tracks ownership would provide a basis for PI as
>>well as further instrumentation. This may not be an
>>issue at the present but I don't think it is too far
>>off.
>>
>>-john
>>
>>
>
>Actually you need to have another kind of semaphore based on a new kind of
>wait-queue: Priority based. I.e. the task with the highest priority get
>woken up first. Then on top of that you build your mutex.
>
That more/less falls out of the PI mechanism. Though it
appears conserving per-mutex data footprint and O(1)
behavior are going to be at odds.

>I was planning to start to look at it and try to see if I could get
>anything to work, but I must admit I haven't got much further than
>just getting Igno's -U8.1 up running.
>
I myself wonder whether Ingo is 1 or N people.

>To get a mutex with priority inheritance add an element pointing to the
>current owner and a field where you store the owners original priority
>which it has to be set back to when it relases the mutex (I am not sure
>how this will work out if someone holds several mutexes!)
>
A task holding several mutexes shouldn't be an issue.
Though per task an ownership list needs to be maintained
in descending priority order such that the effective PI
can be resolved from all task owned mutexes.

Also a multiple ownership model is needed for the case of
shared-reader locks. This results in a list of owners
where the list element can maintain per-mutex task ownership
as well as per-task mutex ownerships.

It is tempting to re-implement the wheel here but existing
works are well on their way:

http://developer.osdl.org/dev/robustmutexes

-john

-- 
john.cooper@timesys.com

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Eugeny S. Mints]

john cooper wrote:
> Esben Nielsen wrote:
> 
>> On Thu, 21 Oct 2004, john cooper wrote:
>>
>>> Mutexes layered on existing semaphores seems convenient
>>> at the moment. However a more native mutex mechanism
>>> which tracks ownership would provide a basis for PI as
>>> well as further instrumentation. This may not be an
>>> issue at the present but I don't think it is too far
>>> off.
>>>
>>> -john
>>>
>>>
>>
>> Actually you need to have another kind of semaphore based on a new 
>> kind of
>> wait-queue: Priority based. I.e. the task with the highest priority get
>> woken up first. Then on top of that you build your mutex.
>>
> That more/less falls out of the PI mechanism. Though it
> appears conserving per-mutex data footprint and O(1)
> behavior are going to be at odds.
> 
>> I was planning to start to look at it and try to see if I could get
>> anything to work, but I must admit I haven't got much further than
>> just getting Igno's -U8.1 up running.
>>
> I myself wonder whether Ingo is 1 or N people.
> 
>> To get a mutex with priority inheritance add an element pointing to the
>> current owner and a field where you store the owners original priority
>> which it has to be set back to when it relases the mutex (I am not sure
>> how this will work out if someone holds several mutexes!)
>>
> A task holding several mutexes shouldn't be an issue.
> Though per task an ownership list needs to be maintained
> in descending priority order such that the effective PI
> can be resolved from all task owned mutexes.

Seems it is too coplex model at least for the first step. The one of 
possible trade-offs coming on mind is to trace the number of resources 
(mutexes) held by a process and to restore original priority only when 
resource count reaches 0. This is one of the sollutions accepted by RTOS 
guys.

The other concern with PI is that most likely PI should be prohibited 
for utilization with locks which are used in the way similar to "waiting 
completition" - i.e. if PI is employed on a mutex it is prohibited to 
release it if a process is not the owner of the mutex.

> 
> Also a multiple ownership model is needed for the case of
> shared-reader locks. This results in a list of owners
> where the list element can maintain per-mutex task ownership
> as well as per-task mutex ownerships.
> It is tempting to re-implement the wheel here but existing
> works are well on their way:
> 
> http://developer.osdl.org/dev/robustmutexes

It is definitly non-trivial work to adapt this approach - there are a 
lot of issues.

				Eugeny
> -john
> 

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[john cooper]

Eugeny S. Mints wrote:

> john cooper wrote:
>
>> A task holding several mutexes shouldn't be an issue.
>> Though per task an ownership list needs to be maintained
>> in descending priority order such that the effective PI
>> can be resolved from all task owned mutexes.
>
>
> Seems it is too coplex model at least for the first step. The one of 
> possible trade-offs coming on mind is to trace the number of resources 
> (mutexes) held by a process and to restore original priority only when 
> resource count reaches 0. This is one of the sollutions accepted by 
> RTOS guys.

It would seem a mutex ownership list still needs to be maintained.
Doing so in unordered priority will give a small fixed insertion
time, but will require an exhaustive search in order to calculate
maximum priority. Doing so in priority order will require an
average of #mutex_owned / 2 for the insertion, and gives a fixed
time for maximum priority calculation. The latter appears to offer
a performance benefit to the degree the incoming priorities are
random.

> The other concern with PI is that most likely PI should be prohibited 
> for utilization with locks which are used in the way similar to 
> "waiting completition" - i.e. if PI is employed on a mutex it is 
> prohibited to release it if a process is not the owner of the mutex.

Yes, that type of usage breaks the notion of ownership.
It would be a error for a task to attempt relinquishing
a mutex which it had not acquired and was holding at the
time of unlock.

>> http://developer.osdl.org/dev/robustmutexes
>
>
> It is definitly non-trivial work to adapt this approach - there are a 
> lot of issues.

I should have qualified that reference. Those folks are
addressing more than PI mutexes. Indeed their goal is
support of fast user mutexes which offer detection of mutex
owners gone astray (exited, killed, etc..). It is the kernel
component of the work to which I was referring.

-john


-- 
john.cooper@timesys.com

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Scott Wood]

On Thu, Oct 21, 2004 at 12:49:30PM -0400, john cooper wrote:
> It would seem a mutex ownership list still needs to be maintained.
> Doing so in unordered priority will give a small fixed insertion
> time, but will require an exhaustive search in order to calculate
> maximum priority. Doing so in priority order will require an
> average of #mutex_owned / 2 for the insertion, and gives a fixed
> time for maximum priority calculation. The latter appears to offer
> a performance benefit to the degree the incoming priorities are
> random.

If you keep it in priority order, then you're paying the O(n) cost
every time you acquire a lock.  If you keep it unordered and only
search it when you need to recalculate a task's priority after a lock
has been released (or priorities have been changed), you pay the cost
much less often.  Plus, the number of locks held by any given thread
should generally be very small.

-Scott

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[john cooper]

Scott Wood wrote:

>On Thu, Oct 21, 2004 at 12:49:30PM -0400, john cooper wrote:
>
>>It would seem a mutex ownership list still needs to be maintained.
>>Doing so in unordered priority will give a small fixed insertion
>>time, but will require an exhaustive search in order to calculate
>>maximum priority. Doing so in priority order will require an
>>average of #mutex_owned / 2 for the insertion, and gives a fixed
>>time for maximum priority calculation. The latter appears to offer
>>a performance benefit to the degree the incoming priorities are
>>random.
>>
>
>If you keep it in priority order, then you're paying the O(n) cost
>every time you acquire a lock.  If you keep it unordered and only
>search it when you need to recalculate a task's priority after a lock
>has been released (or priorities have been changed), you pay the cost
>much less often.
>
That's true for the case where the current priority is
somewhere else handy (likely) and we don't need to traverse
the list for other reasons such as allowing/disallowing
recursive acquisition of a mutex by a given task.

>Plus, the number of locks held by any given thread
>should generally be very small.
>
I agree, it is likely in the noise. The list length and
manipulation of the mutex waiter list overshadows this.

-john



-- 
john.cooper@timesys.com

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Scott Wood]

On Thu, Oct 21, 2004 at 02:09:19PM -0400, john cooper wrote:
> Scott Wood wrote:
> >If you keep it in priority order, then you're paying the O(n) cost
> >every time you acquire a lock. 

I partially take this back; depending on how it's implemented, you
can get away with only adding it to the list once contention occurs.

> That's true for the case where the current priority is
> somewhere else handy (likely) and we don't need to traverse
> the list for other reasons such as allowing/disallowing
> recursive acquisition of a mutex by a given task.

How would maintaining priority order make it faster to check for
recursive usage?  You'd be looking for a specific mutex rather than
the highest priority blocker.  You could also check the per-mutex
list of owners (which you'll need to implement PI on rwlocks), to
avoid needing to add to the locks-held list in non-contended cases.

On uniprocessor, one may wish to turn rwlocks into recursive non-rw
mutexes, where recursion checking would use a single owner field.

Also, keeping it in priority order would introduce yet another place
that assumes of a linear priority scheme.  At some point, it may be
desireable to implement other schemes, such as maintaining per-CPU
priorities to deal with inheriting from CPU-bound tasks without
introducing said tasks' priorities on other CPUs.

-Scott

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[john cooper]

Scott Wood wrote:

>On Thu, Oct 21, 2004 at 02:09:19PM -0400, john cooper wrote:
>
>>That's true for the case where the current priority is
>>somewhere else handy (likely) and we don't need to traverse
>>the list for other reasons such as allowing/disallowing
>>recursive acquisition of a mutex by a given task.
>>
>
>How would maintaining priority order make it faster to check for
>recursive usage?  
>
It wouldn't. My point was an exhaustive traversal may be
needed for other reasons with an insertion sort being
near free.

Yet considering the cost to maintain these lists in priority
order with multiple spinlock acquisition sequences due to how
the aggregate data structure must be traversed/ordered,
I haven't yet convinced myself either way.

>On uniprocessor, one may wish to turn rwlocks into recursive non-rw
>mutexes, where recursion checking would use a single owner field.
>
It isn't obvious to me how this would address the case of a
task holding a reader lock on mx-A then blocking on mx-B.
Another task attempting to acquire a reader lock on mx-A would
block rather than immediately acquiring the lock.

-john


-- 
john.cooper@timesys.com

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Scott Wood]

On Thu, Oct 21, 2004 at 04:18:12PM -0400, john cooper wrote:
> Scott Wood wrote:
> >How would maintaining priority order make it faster to check for
> >recursive usage?  
> >
> It wouldn't. My point was an exhaustive traversal may be
> needed for other reasons with an insertion sort being
> near free.
> 
> Yet considering the cost to maintain these lists in priority
> order with multiple spinlock acquisition sequences due to how
> the aggregate data structure must be traversed/ordered,
> I haven't yet convinced myself either way.

Another issue is that if you keep them in order, you have to fix the
list whenever an owner of a listed mutex changes its priority.

> >On uniprocessor, one may wish to turn rwlocks into recursive non-rw
> >mutexes, where recursion checking would use a single owner field.
> >
> It isn't obvious to me how this would address the case of a
> task holding a reader lock on mx-A then blocking on mx-B.
> Another task attempting to acquire a reader lock on mx-A would
> block rather than immediately acquiring the lock.

Yes.  However, the contention case should not be optimized at the
expense of the common case, which can be faster for non-rwlock
implementations when PI is involved.  On SMP, you'd be introducing a
bottleneck by taking away rwlocks, but on UP it's only an issue when
you get preempted or block in a critical section.

There could be problems if some code tries to acquire read locks
out-of-order, believing that it can't deadlock that way (if the
writers don't nest), but that's a problem anyway unless there's a
reasonable way of implementing PI without limiting the number of
concurrent readers (they have to be stored somewhere, and the
alternatives of setting a hard limit on mutexes-per-thread or doing
dynamic allocation inside the lock function are worse).

-Scott

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[john cooper]

Scott Wood wrote:

>On Thu, Oct 21, 2004 at 04:18:12PM -0400, john cooper wrote:
>
>>Yet considering the cost to maintain these lists in priority
>>order with multiple spinlock acquisition sequences due to how
>>the aggregate data structure must be traversed/ordered,
>>I haven't yet convinced myself either way.
>>
>
>Another issue is that if you keep them in order, you have to fix the
>list whenever an owner of a listed mutex changes its priority.
>
Yes, but my concern was having to backoff in out-of-sequence
spinlock acquisition paths. Looking at it again if the canonical
lock acquisition sequence is a task's mutex-owned list then a
mutex's task-owned list, the nondeterministic backoff (if any)
gets pushed to the case of a waiter blocking on the lock.

>>It isn't obvious to me how this would address the case of a
>>task holding a reader lock on mx-A then blocking on mx-B.
>>Another task attempting to acquire a reader lock on mx-A would
>>block rather than immediately acquiring the lock.
>>
>
>Yes.  However, the contention case should not be optimized at the
>expense of the common case, which can be faster for non-rwlock
>implementations when PI is involved.  On SMP, you'd be introducing a
>bottleneck by taking away rwlocks, but on UP it's only an issue when
>you get preempted or block in a critical section.
>
My concern is removing what should be available reader
concurrency for the mutex in question. I can't assess
how un/common that may be over all application scenarios.

-john


-- 
john.cooper@timesys.com

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Scott Wood]

On Thu, Oct 21, 2004 at 06:15:00PM -0400, john cooper wrote:
> Yes, but my concern was having to backoff in out-of-sequence
> spinlock acquisition paths. 

Out-of-sequence acquisition is a bug, unless the caller uses trylocks
and handles backoff itself.

-Scott

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[john cooper]

Scott Wood wrote:

>On Thu, Oct 21, 2004 at 06:15:00PM -0400, john cooper wrote:
>
>>Yes, but my concern was having to backoff in out-of-sequence
>>spinlock acquisition paths. 
>>
>
>Out-of-sequence acquisition is a bug, unless the caller uses trylocks
>and handles backoff itself.
>
Understood -- we may be getting hung up on terminology here.

Rather the issue was whether the nondeterministic out-of-sequence
backoff could be pushed to a noncritical path. I believe so.
It is further likely a backoff would not be needed as the
a path acquiring a mutex's task-owned list lock during a
priority promotion scan shouldn't have reason to acquire any
task's mutex-owned list lock. The latter list would only need
to be locked at time of successful mutex acquisition/free.

-john

-- 
john.cooper@timesys.com

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Esben Nielsen]

Esben Nielsen
Work:
 Cotas Computer Technology A/S
 Paludan Mullersvej 82
 8200 Aarhus N 
Private
 Moellegade 7A, 3., 4
 8000 Aarhus C
Phone: 
 +45 86 12 73 79
Mobile:
 +45 27 13 10 05


On Thu, 21 Oct 2004, Scott Wood wrote:

> On Thu, Oct 21, 2004 at 02:09:19PM -0400, john cooper wrote:
> > Scott Wood wrote:
> > >If you keep it in priority order, then you're paying the O(n) cost
> > >every time you acquire a lock. 
> 
> I partially take this back; depending on how it's implemented, you
> can get away with only adding it to the list once contention occurs.
>

You can implement the full scheduler structure in each mutex. That would
be O(1) but would take take quite a lot of memory.

On the other hand a sorted list will not take more memory than now but
will appear to be O(n) when inserting into the list. However, on a UP
machine no lower priority task will run when higher priority tasks runs.
I.e. you will always add to the beginning of the list. That is assuming
ofcourse that nobody will sleep while holding a mutex - which is a bad
bahaviour. And if you try to take another mutex, while holding one already
and you have to block, the holder of the second mutex will be moved up to
your priority. I.e. it will block the CPU for any lower priority task...

I don't know what it would be on SMP systems though...

 
> > That's true for the case whe~re the current priority is
> > somewhere else handy (likely) and we don't need to traverse
> > the list for other reasons such as allowing/disallowing
> > recursive acquisition of a mutex by a given task.
> 
> How would maintaining priority order make it faster to check for
> recursive usage?  You'd be looking for a specific mutex rather than
> the highest priority blocker.  You could also check the per-mutex
> list of owners (which you'll need to implement PI on rwlocks), to
> avoid needing to add to the locks-held list in non-contended cases.
> 
> On uniprocessor, one may wish to turn rwlocks into recursive non-rw
> mutexes, where recursion checking would use a single owner field.
>


Why not use a simple counter? The mutex protects it's own memory area.
Anyway, I am not sure recursive acquisition is such a good idea: It will
make the mutex more expensive to use and promote sloppy coding where you
don't really know what mutexes are held right now. When you are building a
subsystem you can send around a flag in the argument saying whether you
have taken the lock or not. If you call into other systems, where you
can't add a parameter to each function, you should release your mutex(es)
anyway! I think it is better that the sloppy coder discoveres that he
deadlocks on himself before getting other sub-systems involved :-)

> Also, keeping it in priority order would introduce yet another place
> that assumes of a linear priority scheme.  At some point, it may be
> desireable to implement other schemes, such as maintaining per-CPU
> priorities to deal with inheriting from CPU-bound tasks without
> introducing said tasks' priorities on other CPUs.
>

I am not sure this at all make sense on a SMP system. For performance
reasons I think that one should stick to ordinary semaphores and in a lot
of places spin-locks on such a system. Even with a dedicated RTOS you have
to design your system from the bottum up to get a good real-time
behaviour - and it depends a lot on the application of which you can only
have one! That is very far from the world of SMP servers.

The ones who can use all the fancy mutexes are really embedded developer
(like myself) who need a robust RTOS but also drivers for a lot of
hardware, a good TCP/IP stack, firewall, good filesystems etc. which the
commercial vendors have a hard time delivering all at once. 

On the desktop it can lead to good performance of multimedia but as soon
as you want to use two applications, which considers themselves multimedia
and thus gives themselves high priority, it wont work.

I must also say, that from my perspective low latencies is not the issue.
The issue is predictability: I must be able to create threads and know
they can't all the sudden be preempted by all kinds of things. I.e. if I
give them higher priority than all the "normal" stuff and all shared
resources between my tasks and the "normal" stuff are locked with mutexes
using prirority inheritance and only for fixed amount of time, I am in the
clear. It is also important that all interrupts and spin-locks are only
held for a fixed amount of time - but as long as that time is lower than
the maximum latency and is bounded in occurance I don't really
care. Forinstance a driver servicing a serial channel wont hurt being run
in interrupt context as it is really limited how much CPU it can take
overall.

> -Scott

Esben

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Scott Wood]

On Thu, Oct 21, 2004 at 11:01:21PM +0200, Esben Nielsen wrote:
> You can implement the full scheduler structure in each mutex. That would
> be O(1) but would take take quite a lot of memory.

Are you talking about the thread's locks-held list, or the mutex's
blocked-threads list?  In the latter case, you could cut down on the
memory usage by allocating one such structure for each thread upon
creation, placing it into a pool, and associating them with a mutex
when it's contended (as you can't have more contended mutexes than
you have threads).  It's still a lot heavier than a linked list,
though.

In either case, you'd need to do whatever adjustments to the
scheduler's data structures are necessary whenever a task's priority
changes (including via PI).  It's worth it for at least one of the
lists, as if neither locks-held nor threads-blocked is kept in order,
priority recalculation becomes O(n^2) to find the highest priority
blocker among all held mutexes.  Ordering threads-blocked seems to
make more sense, as you don't have the imbalance between the
frequency adding to the list and searching the list.

> On the other hand a sorted list will not take more memory than now but
> will appear to be O(n) when inserting into the list. However, on a UP
> machine no lower priority task will run when higher priority tasks runs.
> I.e. you will always add to the beginning of the list. That is assuming
> ofcourse that nobody will sleep while holding a mutex - which is a bad
> bahaviour.

It's bad, but inevitable if this is also used to provide PI mutexes
to userspace.

> > On uniprocessor, one may wish to turn rwlocks into recursive non-rw
> > mutexes, where recursion checking would use a single owner field.
> >
> 
> 
> Why not use a simple counter? 

You'd need a counter as well, but you first have to check the owner
to make sure that the count reflects the calling thread's activity
rather than some other thread.

> The mutex protects it's own memory area.
> Anyway, I am not sure recursive acquisition is such a good idea: It will
> make the mutex more expensive to use and promote sloppy coding where you
> don't really know what mutexes are held right now.

I agree, but current code assumes rwlocks can be recursively obtained
for reading (unless that's changed recently).

> I am not sure this at all make sense on a SMP system. For performance
> reasons I think that one should stick to ordinary semaphores and in a lot
> of places spin-locks on such a system. Even with a dedicated RTOS you have
> to design your system from the bottum up to get a good real-time
> behaviour - and it depends a lot on the application of which you can only
> have one! That is very far from the world of SMP servers.

Things get weird on SMP, but it's possible to produce reasonable
real-time behavior, and there are people out there who are interested
in it.

> The ones who can use all the fancy mutexes are really embedded developer
> (like myself) who need a robust RTOS but also drivers for a lot of
> hardware, a good TCP/IP stack, firewall, good filesystems etc. which the
> commercial vendors have a hard time delivering all at once. 

And some embedded devices also need a lot of CPU power, and some of
those use SMP.

> On the desktop it can lead to good performance of multimedia but as soon
> as you want to use two applications, which considers themselves multimedia
> and thus gives themselves high priority, it wont work.

It can be made to work if you have CPU reservations, or some other
way of ensuring that the applications take their CPU in
appropriately sized chunks.

-Scott

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[john cooper]

Esben Nielsen wrote:

>Anyway, I am not sure recursive acquisition is such a good idea: It will
>make the mutex more expensive to use and promote sloppy coding where you
>don't really know what mutexes are held right now.
>
In my experience I haven't quite found this to be the case.
Rather allowing a lock/mutex/etc.. to be recursively acquired
within a given path can greatly simplify APIs. Such as
cases where a primitive acquires a lock internally which is
also know by and suited to protect data in the caller's context.
This avoids the "who has the lock?" information which must
otherwise get stuffed into the API. Nested function calls
simply observe correct lock usage at their respective levels
and all unwinds correctly.

The one notable place where this doesn't work is where
locks must be conditionally acquired in an out-of-order sequence.
However often these cases can be pushed out of the externally
visible API. Another operational consideration is maintaining
debug information in the lock. Keeping track of where each
lock acquisition was made is at odds with a fixed sized lock
data footprint. Still recursive acquisitions scale only
with available stack space so a fair upper limit approximation
is possible when running on fixed size stacks. Note I'm
refering to single-owner spinlocks/mutexes here rather than
reader/write locks.

>I think it is better that the sloppy coder discoveres that he
>deadlocks on himself before getting other sub-systems involved :-)
>
I agree. But I don't think the above precludes doing so.
Detecting violations of unbalanced lock/unlock calls isn't
really different than for non-recursive primitives.

-john

-- 
john.cooper@timesys.com

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Eugeny S. Mints]

Scott Wood wrote:
> On Thu, Oct 21, 2004 at 12:49:30PM -0400, john cooper wrote:
> 
>>It would seem a mutex ownership list still needs to be maintained.
>>Doing so in unordered priority will give a small fixed insertion
>>time, but will require an exhaustive search in order to calculate
>>maximum priority. Doing so in priority order will require an
>>average of #mutex_owned / 2 for the insertion, and gives a fixed
>>time for maximum priority calculation. The latter appears to offer
>>a performance benefit to the degree the incoming priorities are
>>random.
> 
> 
> If you keep it in priority order, then you're paying the O(n) cost
> every time you acquire a lock.  If you keep it unordered and only
> search it when you need to recalculate a task's priority after a lock
> has been released (or priorities have been changed), you pay the cost
> much less often.  Plus, the number of locks held by any given thread
> should generally be very small.
As to locks held by any given thread - it's not always true - take a 
look at mm/filemap.c locks nesting map in comments.

                        Eugeny

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Scott Wood]

On Thu, Oct 21, 2004 at 10:10:17PM +0400, Eugeny S. Mints wrote:
> Scott Wood wrote:
> >If you keep it in priority order, then you're paying the O(n) cost
> >every time you acquire a lock.  If you keep it unordered and only
> >search it when you need to recalculate a task's priority after a lock
> >has been released (or priorities have been changed), you pay the cost
> >much less often.  Plus, the number of locks held by any given thread
> >should generally be very small.
> As to locks held by any given thread - it's not always true - take a 
> look at mm/filemap.c locks nesting map in comments.

I guess it depends on the definition of "very small" and "generally".
:-)

A nesting of 5 locks is pushing the limits of "very small", but it's
still no big deal to iterate over once in a while.

-Scott

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Eugeny S. Mints]

john cooper wrote:
> Eugeny S. Mints wrote:
> 
>> john cooper wrote:
>>
>>> A task holding several mutexes shouldn't be an issue.
>>> Though per task an ownership list needs to be maintained
>>> in descending priority order such that the effective PI
>>> can be resolved from all task owned mutexes.
>>
>>
>>
>> Seems it is too coplex model at least for the first step. The one of 
>> possible trade-offs coming on mind is to trace the number of resources 
>> (mutexes) held by a process and to restore original priority only when 
>> resource count reaches 0. This is one of the sollutions accepted by 
>> RTOS guys.
> 
> 
> It would seem a mutex ownership list still needs to be maintained.
> Doing so in unordered priority will give a small fixed insertion
> time, but will require an exhaustive search in order to calculate
> maximum priority. Doing so in priority order will require an
> average of #mutex_owned / 2 for the insertion, and gives a fixed
> time for maximum priority calculation. The latter appears to offer
> a performance benefit to the degree the incoming priorities are
> random.

Yes, definilty, I 100% agree with you - I have _not_ disputed the 
priority list approach at all.

>> The other concern with PI is that most likely PI should be prohibited 
>> for utilization with locks which are used in the way similar to 
>> "waiting completition" - i.e. if PI is employed on a mutex it is 
>> prohibited to release it if a process is not the owner of the mutex.
> 
> 
> Yes, that type of usage breaks the notion of ownership.
> It would be a error for a task to attempt relinquishing
> a mutex which it had not acquired and was holding at the
> time of unlock.

exactly

>>> http://developer.osdl.org/dev/robustmutexes
>>
>>
>>
>> It is definitly non-trivial work to adapt this approach - there are a 
>> lot of issues.
> 
> 
> I should have qualified that reference. Those folks are
> addressing more than PI mutexes. Indeed their goal is
> support of fast user mutexes which offer detection of mutex
> owners gone astray (exited, killed, etc..). It is the kernel
> component of the work to which I was referring.

Ok, I've also talked about kernel component not user space one. User 
space approach with robustness, fast uncontented obtaining, etc looks 
very attractive but both kernel and user space parts have their issues -
though I took a glance to the project about 2 months ago.

			Eugeny

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Scott Wood]

On Thu, Oct 21, 2004 at 07:47:29PM +0400, Eugeny S. Mints wrote:
> Seems it is too coplex model at least for the first step. The one of 
> possible trade-offs coming on mind is to trace the number of resources 
> (mutexes) held by a process and to restore original priority only when 
> resource count reaches 0. This is one of the sollutions accepted by RTOS 
> guys.

That complicates analysis, though, since you now have to look at all
critical sections that the shared-with-high-priority-threads critical
sections nest in.  IMHO, it's important that the inherited priority
be given up as soon as the resource is released.

-Scott

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Jens Axboe]

On Thu, Oct 21 2004, Thomas Gleixner wrote:
> On Thu, 2004-10-21 at 12:11, Jens Axboe wrote:
> > On Thu, Oct 21 2004, Thomas Gleixner wrote:
> > > On Thu, 2004-10-21 at 11:53, Jens Axboe wrote:
> > > > On Thu, Oct 21 2004, Thomas Gleixner wrote:
> > > > > On Thu, 2004-10-21 at 11:12, Rui Nuno Capela wrote:
> > > > > >  [<e018e139>] queuecommand+0x70/0x7c [usb_storage] (24)
> > > > > 
> > > > > As I already pointed out, this is a problem due to up(sema) in
> > > > > queuecommand. That's one of the semaphore abuse points, which needs to
> > > > > be fixed. 
> > > > > 
> > > > > The problem is that semaphores are hold by Process A and released by
> > > > > Process B, which makes Ingo's checks trigger
> > > > 
> > > > That's utter crap, it's perfectly valid use.
> > > 
> > > It's not!
> > > 
> > > >From the code:
> > > 
> > >          init_MUTEX_LOCKED(&(us->sema));
> > > 
> > > This is used to wait for command completion and therefor we have the
> > > completion API. It was used this way because the ancestor of completion
> > > (sleep_on) was racy !
> > 
> > I didn't look at the USB code, I'm just saying that it's perfectly valid
> > use of a semaphore the pattern you describe (process A holding it,
> > process B releasing it).
> 
> Yeah, for a semaphore it is, but not for a mutex.
> 
> IMHO, this is not clearly seperated and therefor produces a lot of
> confusion.

Semaphore and mutex has always been the same thing in Linux. Apparently
this isn't so in Ingos tree, you should make a clear distinction on
which you are discussing.

-- 
Jens Axboe

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Thomas Gleixner]

On Thu, 2004-10-21 at 13:11, Jens Axboe wrote:
> > 
> > IMHO, this is not clearly seperated and therefor produces a lot of
> > confusion.
> 
> Semaphore and mutex has always been the same thing in Linux. Apparently
> this isn't so in Ingos tree, you should make a clear distinction on
> which you are discussing.

I agree, but this thread is discussing Ingo's tree :)

I know that semaphores and mutexes are the same, but that's something
which should be seperated IMHO. 

Ingo's changes reviel a couple of places where completion or wait_event
is more clean, than using a mutex. That's all I'm talking about. Sorry,
if I did not express it clearly enough or even used a wrong expression. 

The points, which we identify are not wrong from the view point when
they were coded. They use a mutex to wait for completion, which is
functional by the mutex implementation and common use in the kernel.

Part of this discussion is the given mixup in the kernel, which is
functionally correct, but if a mutex is changed to a real mutex then it
is wrong in the semantical sense.

tglx

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

* Jens Axboe <axboe@suse.de> wrote:

> I didn't look at the USB code, I'm just saying that it's perfectly
> valid use of a semaphore the pattern you describe (process A holding
> it, process B releasing it).

yes, that is perfectly true, and sorry if we gave you the wrong
impression.

the goal of these patches is to do a semaphore->completion conversion in
cases where the semaphore was used for completion purposes. It's a bit
faster and more readable but not a 'bugfix' in any way. (another set of
patches are converting sleep_on() uses to wait_event*() plus waitqueues
- those can in fact be considered bugfixes in some cases.)

typically the cases where semaphores are held by one task and released
by another task happens coincide with this used-for-completion scenario.

[ the different-owner assert that triggers in my PREEMPT_REALTIME tree
is for completely different reasons and has no impact on upstream at
all. (It merely means 'Ingo does some weird stuff again, pester him, not
others'.) ]

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Jens Axboe]

On Thu, Oct 21 2004, Ingo Molnar wrote:
> 
> * Jens Axboe <axboe@suse.de> wrote:
> 
> > I didn't look at the USB code, I'm just saying that it's perfectly
> > valid use of a semaphore the pattern you describe (process A holding
> > it, process B releasing it).
> 
> yes, that is perfectly true, and sorry if we gave you the wrong
> impression.
> 
> the goal of these patches is to do a semaphore->completion conversion in
> cases where the semaphore was used for completion purposes. It's a bit
> faster and more readable but not a 'bugfix' in any way. (another set of
> patches are converting sleep_on() uses to wait_event*() plus waitqueues
> - those can in fact be considered bugfixes in some cases.)
> 
> typically the cases where semaphores are held by one task and released
> by another task happens coincide with this used-for-completion scenario.

Thanks for the explanation, I can agree with that.

-- 
Jens Axboe

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Bill Huey]

On Thu, Oct 21, 2004 at 12:11:03PM +0200, Jens Axboe wrote:
> I didn't look at the USB code, I'm just saying that it's perfectly valid
> use of a semaphore the pattern you describe (process A holding it,
> process B releasing it).

A lot of things are perfectly "valid" in the Linux kernel regarding
stuff like that are a bit irregular. But the preemption work about
to stress these things in ways that was never designed to which is
why these patches are needed. Having a clear use of various locking
conventions is key to getting this system to behave in a predictable
manner. Quite simply, Linux was never targetted to do this and the
sloppiness is showing so it's got to be removed.

bill

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Jens Axboe]

On Thu, Oct 21 2004, Bill Huey wrote:
> On Thu, Oct 21, 2004 at 12:11:03PM +0200, Jens Axboe wrote:
> > I didn't look at the USB code, I'm just saying that it's perfectly valid
> > use of a semaphore the pattern you describe (process A holding it,
> > process B releasing it).
> 
> A lot of things are perfectly "valid" in the Linux kernel regarding
> stuff like that are a bit irregular. But the preemption work about
> to stress these things in ways that was never designed to which is
> why these patches are needed. Having a clear use of various locking
> conventions is key to getting this system to behave in a predictable
> manner. Quite simply, Linux was never targetted to do this and the
> sloppiness is showing so it's got to be removed.

I have to disagree, I don't think the above use is either convoluted or
sloppy in any way. Now that we have the completion structure, certain
things are surely better implemented as such. But the old use is
perfectly valid and logical, imho.

-- 
Jens Axboe

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Bill Huey]

On Thu, Oct 21, 2004 at 10:14:43PM +0200, Jens Axboe wrote:
> On Thu, Oct 21 2004, Bill Huey wrote:
> > A lot of things are perfectly "valid" in the Linux kernel regarding
> > stuff like that are a bit irregular. But the preemption work about
> > to stress these things in ways that was never designed to which is
> > why these patches are needed. Having a clear use of various locking
> > conventions is key to getting this system to behave in a predictable
> > manner. Quite simply, Linux was never targetted to do this and the
> > sloppiness is showing so it's got to be removed.
> 
> I have to disagree, I don't think the above use is either convoluted or
> sloppy in any way. Now that we have the completion structure, certain
> things are surely better implemented as such. But the old use is
> perfectly valid and logical, imho.

You use a semaphore to protect data, a completion isn't protecting data
but preserving a certain kind of wait ordering in the code. The
possibility of overloading the current mutex_t for PI makes for a conceptual
mismatch when used in this case since having a kind of priority for
completions is a bit odd. It's better to flat out use a completion
instead, IMO.

bill

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Jens Axboe]

On Thu, Oct 21 2004, Bill Huey wrote:
> On Thu, Oct 21, 2004 at 10:14:43PM +0200, Jens Axboe wrote:
> > On Thu, Oct 21 2004, Bill Huey wrote:
> > > A lot of things are perfectly "valid" in the Linux kernel regarding
> > > stuff like that are a bit irregular. But the preemption work about
> > > to stress these things in ways that was never designed to which is
> > > why these patches are needed. Having a clear use of various locking
> > > conventions is key to getting this system to behave in a predictable
> > > manner. Quite simply, Linux was never targetted to do this and the
> > > sloppiness is showing so it's got to be removed.
> > 
> > I have to disagree, I don't think the above use is either convoluted or
> > sloppy in any way. Now that we have the completion structure, certain
> > things are surely better implemented as such. But the old use is
> > perfectly valid and logical, imho.
> 
> You use a semaphore to protect data, a completion isn't protecting data
> but preserving a certain kind of wait ordering in the code. The
> possibility of overloading the current mutex_t for PI makes for a conceptual
> mismatch when used in this case since having a kind of priority for
> completions is a bit odd. It's better to flat out use a completion
> instead, IMO.

Linux semaphores (being counted) have always been a fine fit for things
like the loop use, where you get to down it 10 times because you have 10
items pending. I know this isn't the traditional mutex and that it
doesn't protect data as such, but is was never abuse. It isn't overload.
Doing it with a traditional mutex (I'm assuming this is what mutex_t is
in Ingos tree) would be overload and a bad idea, indeed.

-- 
Jens Axboe

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Bill Huey]

On Thu, Oct 21, 2004 at 10:33:50PM +0200, Jens Axboe wrote:
> On Thu, Oct 21 2004, Bill Huey wrote:
> > You use a semaphore to protect data, a completion isn't protecting data
> > but preserving a certain kind of wait ordering in the code. The
> > possibility of overloading the current mutex_t for PI makes for a conceptual
> > mismatch when used in this case since having a kind of priority for
> > completions is a bit odd. It's better to flat out use a completion
> > instead, IMO.
> 
> Linux semaphores (being counted) have always been a fine fit for things
> like the loop use, where you get to down it 10 times because you have 10
> items pending. I know this isn't the traditional mutex and that it
> doesn't protect data as such, but is was never abuse. It isn't overload.
> Doing it with a traditional mutex (I'm assuming this is what mutex_t is
> in Ingos tree) would be overload and a bad idea, indeed.

Well, this is something that's got to be considered by the larger Linux
community and whether these conventions are to be kept or removed. It's
a larger issue than what can be address in Ingo's preemption patch, but
with inevitable need for something like this in the kernel (hard RT)
it's really unavoidable collision. IMO, it's got to go, which is a nasty
change.

bill

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Thomas Gleixner]

On Thu, 2004-10-21 at 22:38, Bill Huey wrote:
> On Thu, Oct 21, 2004 at 10:33:50PM +0200, Jens Axboe wrote:
> > On Thu, Oct 21 2004, Bill Huey wrote:
> > > You use a semaphore to protect data, a completion isn't protecting data
> > > but preserving a certain kind of wait ordering in the code. The
> > > possibility of overloading the current mutex_t for PI makes for a conceptual
> > > mismatch when used in this case since having a kind of priority for
> > > completions is a bit odd. It's better to flat out use a completion
> > > instead, IMO.
> > 
> > Linux semaphores (being counted) have always been a fine fit for things
> > like the loop use, where you get to down it 10 times because you have 10
> > items pending. I know this isn't the traditional mutex and that it
> > doesn't protect data as such, but is was never abuse. It isn't overload.
> > Doing it with a traditional mutex (I'm assuming this is what mutex_t is
> > in Ingos tree) would be overload and a bad idea, indeed.
> 
> Well, this is something that's got to be considered by the larger Linux
> community and whether these conventions are to be kept or removed. It's
> a larger issue than what can be address in Ingo's preemption patch, but
> with inevitable need for something like this in the kernel (hard RT)
> it's really unavoidable collision. IMO, it's got to go, which is a nasty
> change.
> 

Hey, let's stop this here.

You are both (in)correct :)

1. It makes no sense to discuss, why X has been considered correct for
time T.

2. Counted semaphores are a valid use and should be marked explicit as
counted semaphores.

3. Using mutexes and semaphores for event and completion signalling
should be converted to the appropriate interfaces. 

A bunch of work, but not really hard.

tglx

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Bill Huey]

On Thu, Oct 21, 2004 at 10:43:41PM +0200, Thomas Gleixner wrote:
> Hey, let's stop this here.
> 
> You are both (in)correct :)
> 
> 1. It makes no sense to discuss, why X has been considered correct for
> time T.
> 
> 2. Counted semaphores are a valid use and should be marked explicit as
> counted semaphores.
> 
> 3. Using mutexes and semaphores for event and completion signalling
> should be converted to the appropriate interfaces. 
> 
> A bunch of work, but not really hard.

What's the verdict ? leave the lock detector alone or change it ?

bill

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Jens Axboe]

On Thu, Oct 21 2004, Thomas Gleixner wrote:
> On Thu, 2004-10-21 at 22:38, Bill Huey wrote:
> > On Thu, Oct 21, 2004 at 10:33:50PM +0200, Jens Axboe wrote:
> > > On Thu, Oct 21 2004, Bill Huey wrote:
> > > > You use a semaphore to protect data, a completion isn't protecting data
> > > > but preserving a certain kind of wait ordering in the code. The
> > > > possibility of overloading the current mutex_t for PI makes for a conceptual
> > > > mismatch when used in this case since having a kind of priority for
> > > > completions is a bit odd. It's better to flat out use a completion
> > > > instead, IMO.
> > > 
> > > Linux semaphores (being counted) have always been a fine fit for things
> > > like the loop use, where you get to down it 10 times because you have 10
> > > items pending. I know this isn't the traditional mutex and that it
> > > doesn't protect data as such, but is was never abuse. It isn't overload.
> > > Doing it with a traditional mutex (I'm assuming this is what mutex_t is
> > > in Ingos tree) would be overload and a bad idea, indeed.
> > 
> > Well, this is something that's got to be considered by the larger Linux
> > community and whether these conventions are to be kept or removed. It's
> > a larger issue than what can be address in Ingo's preemption patch, but
> > with inevitable need for something like this in the kernel (hard RT)
> > it's really unavoidable collision. IMO, it's got to go, which is a nasty
> > change.
> > 
> 
> Hey, let's stop this here.
> 
> You are both (in)correct :)
> 
> 1. It makes no sense to discuss, why X has been considered correct for
> time T.

Because it is correct. Debating that it's now incorrect because it
inconveniently happens to make some detection scheme harder is silly.

> 2. Counted semaphores are a valid use and should be marked explicit as
> counted semaphores.

Indeed

> 3. Using mutexes and semaphores for event and completion signalling
> should be converted to the appropriate interfaces. 

Agree. Do you test all your conversions? Whole-sale conversions like
this tend to break at least some of the drivers. And that's totally
unacceptable, breaking a working solution because of something that's
not really a bug.

> A bunch of work, but not really hard.

Not if you don't test it.

-- 
Jens Axboe

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Bill Huey]

On Thu, Oct 21, 2004 at 01:38:21PM -0700, Bill Huey wrote:
> On Thu, Oct 21, 2004 at 10:33:50PM +0200, Jens Axboe wrote:
> > Linux semaphores (being counted) have always been a fine fit for things
> > like the loop use, where you get to down it 10 times because you have 10
> > items pending. I know this isn't the traditional mutex and that it
> > doesn't protect data as such, but is was never abuse. It isn't overload.
> > Doing it with a traditional mutex (I'm assuming this is what mutex_t is
> > in Ingos tree) would be overload and a bad idea, indeed.
> 
> Well, this is something that's got to be considered by the larger Linux
> community and whether these conventions are to be kept or removed. It's
> a larger issue than what can be address in Ingo's preemption patch, but
> with inevitable need for something like this in the kernel (hard RT)
> it's really unavoidable collision. IMO, it's got to go, which is a nasty
> change.

But this is a non-fatal case. I'll see if I can change this logic to not
completely die when this case is detected.

bill

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Jens Axboe]

On Thu, Oct 21 2004, Bill Huey wrote:
> On Thu, Oct 21, 2004 at 10:33:50PM +0200, Jens Axboe wrote:
> > On Thu, Oct 21 2004, Bill Huey wrote:
> > > You use a semaphore to protect data, a completion isn't protecting data
> > > but preserving a certain kind of wait ordering in the code. The
> > > possibility of overloading the current mutex_t for PI makes for a conceptual
> > > mismatch when used in this case since having a kind of priority for
> > > completions is a bit odd. It's better to flat out use a completion
> > > instead, IMO.
> > 
> > Linux semaphores (being counted) have always been a fine fit for things
> > like the loop use, where you get to down it 10 times because you have 10
> > items pending. I know this isn't the traditional mutex and that it
> > doesn't protect data as such, but is was never abuse. It isn't overload.
> > Doing it with a traditional mutex (I'm assuming this is what mutex_t is
> > in Ingos tree) would be overload and a bad idea, indeed.
> 
> Well, this is something that's got to be considered by the larger Linux
> community and whether these conventions are to be kept or removed. It's
> a larger issue than what can be address in Ingo's preemption patch, but
> with inevitable need for something like this in the kernel (hard RT)
> it's really unavoidable collision. IMO, it's got to go, which is a nasty
> change.

It has to go, why? Because your deadlock detection breaks? Doesn't seem
a very strong reason to me at all, sorry.

-- 
Jens Axboe

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

* Jens Axboe <axboe@suse.de> wrote:

> It has to go, why? Because your deadlock detection breaks? Doesn't
> seem a very strong reason to me at all, sorry.

no, this is no reason at all. I'm really sorry this issue came up in
this context because now people appear to be arguing this as some sort
of policy issue, implying that is somehow improper to use mutexes
instead of completions, which it clearly is _not_. I very much wanted to
avoid this particular type of flamewar :-)

Using mutexes for completion purposes is perfectly fine kernel code. 
Full stop.

Using completions instead of mutexes in certain cases has some minor
advantages for two simple reasons: it's slighly faster and it's also
more readable.

here's an example: initially i made the scheduler's migration logic use
semaphores in that fashion and Linus made me change it to completions,
because, and i quote Linus here:

 [...]
 Btw, should you not use completions here?

 Completions are optimized for the sleep (ie contention) case, while
 semaphores are optimized for the non-contention case. It also makes 
 more "sense" from a conceptual angle (you're waiting for something to
 complete, not asking for an exclusive thing).
 [...]

and i have to say the migration code did become cleaner. To signal some
sort of event it's a more intuitive _symbol_ _name_ to use 'complete()'
and 'wait_for_completion()' than to use 'up()' and 'down()'.

[ If you truly do not agree with this contention then please just look
at one simple conversion we did and check out the previous and the new
logic, by reading the full previous code and the full resulting code. I
do believe that if anyone at that point still thinks that the
semaphore-based code is just as readable (in that context!) as the
completion-based code that then his brains are not made of neurons but
silicon :) ]

but it has never been kernel policy to not allow the use of mutexes that
way! In some cases it's somewhat cleaner to use completions (and if
something is cleaner in Linux then in most cases it's faster as well),
but it's a judgement thing just like it's judgement thing whether to use
kmalloc() or get_free_pages(). Both are correct for the generic problem
of 'allocate some kernel RAM', but optimized for two different types of
uses.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Jens Axboe]

On Fri, Oct 22 2004, Ingo Molnar wrote:
> 
> * Jens Axboe <axboe@suse.de> wrote:
> 
> > It has to go, why? Because your deadlock detection breaks? Doesn't
> > seem a very strong reason to me at all, sorry.
> 
> no, this is no reason at all. I'm really sorry this issue came up in
> this context because now people appear to be arguing this as some sort
> of policy issue, implying that is somehow improper to use mutexes
> instead of completions, which it clearly is _not_. I very much wanted to
> avoid this particular type of flamewar :-)
> 
> Using mutexes for completion purposes is perfectly fine kernel code. 
> Full stop.
> 
> Using completions instead of mutexes in certain cases has some minor
> advantages for two simple reasons: it's slighly faster and it's also
> more readable.
> 
> here's an example: initially i made the scheduler's migration logic use
> semaphores in that fashion and Linus made me change it to completions,
> because, and i quote Linus here:
> 
>  [...]
>  Btw, should you not use completions here?
> 
>  Completions are optimized for the sleep (ie contention) case, while
>  semaphores are optimized for the non-contention case. It also makes 
>  more "sense" from a conceptual angle (you're waiting for something to
>  complete, not asking for an exclusive thing).
>  [...]
> 
> and i have to say the migration code did become cleaner. To signal some
> sort of event it's a more intuitive _symbol_ _name_ to use 'complete()'
> and 'wait_for_completion()' than to use 'up()' and 'down()'.
> 
> [ If you truly do not agree with this contention then please just look
> at one simple conversion we did and check out the previous and the new
> logic, by reading the full previous code and the full resulting code. I
> do believe that if anyone at that point still thinks that the
> semaphore-based code is just as readable (in that context!) as the
> completion-based code that then his brains are not made of neurons but
> silicon :) ]

I fully agree with everything in your mail so far. What annoyed me is
some people advocating their changes under the false pretense that
existing use was broken, which it isn't.

completions _do_ make cleaner code for the intended case. But your
writing above is very clear and already explains that very well.

Lets put the issue to rest and get back to more productive work!

-- 
Jens Axboe

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

* Jens Axboe <axboe@suse.de> wrote:

> I fully agree with everything in your mail so far. What annoyed me is
> some people advocating their changes under the false pretense that
> existing use was broken, which it isn't.

yeah, and i have to say that such advocacy mostly comes from the natural
desire to solve those _other_ problems that non-standard locking designs
have with Linux mutexes. But those problems are that of the other trees
alone, not upstream's :) Suggesting that those problems are in any way
upstream's problem, even if well-intentioned, can be quite offensive.

> completions _do_ make cleaner code for the intended case. But your
> writing above is very clear and already explains that very well.
> 
> Lets put the issue to rest and get back to more productive work!

/me rejoices :)

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Bill Huey]

On Fri, Oct 22, 2004 at 08:19:01AM +0200, Jens Axboe wrote:
> It has to go, why? Because your deadlock detection breaks? Doesn't seem
> a very strong reason to me at all, sorry.

The deadlock detector is needed. Whether you understand that or not is
irrelevant to the current work that's being done. And your idiot attacks
against it doesn't correct these issues nor does it gain credibility
with the audience that does find it useful.

bill

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Jens Axboe]

On Fri, Oct 22 2004, Bill Huey wrote:
> On Fri, Oct 22, 2004 at 08:19:01AM +0200, Jens Axboe wrote:
> > It has to go, why? Because your deadlock detection breaks? Doesn't seem
> > a very strong reason to me at all, sorry.
> 
> The deadlock detector is needed. Whether you understand that or not is
> irrelevant to the current work that's being done. And your idiot attacks
> against it doesn't correct these issues nor does it gain credibility
> with the audience that does find it useful.

*plonk*

If you can't stand criticism without resorting to feeble personal
attacks, I suggest you go elsewhere.

-- 
Jens Axboe

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Bill Huey]

On Fri, Oct 22, 2004 at 10:59:28AM +0200, Jens Axboe wrote:
> On Fri, Oct 22 2004, Bill Huey wrote:
> > On Fri, Oct 22, 2004 at 08:19:01AM +0200, Jens Axboe wrote:
> > > It has to go, why? Because your deadlock detection breaks? Doesn't seem
> > > a very strong reason to me at all, sorry.
> > 
> > The deadlock detector is needed. Whether you understand that or not is
> > irrelevant to the current work that's being done. And your idiot attacks
> > against it doesn't correct these issues nor does it gain credibility
> > with the audience that does find it useful.
> 
> *plonk*
> 
> If you can't stand criticism without resorting to feeble personal
> attacks, I suggest you go elsewhere.

Then stick to the topic at hand, suggest positive changes, and cut the
crap with implied personal attacks like the above. If you hadn't pull
the discussion to that point, I wouldn't have reacted that way. It's
completely juvenile behavior from you and you can't expect me or
anybody else to take it sitting down.

bill

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Bill Huey]

On Fri, Oct 22, 2004 at 02:06:37AM -0700, Bill Huey wrote:
> On Fri, Oct 22, 2004 at 10:59:28AM +0200, Jens Axboe wrote:
> > *plonk*
> > 
> > If you can't stand criticism without resorting to feeble personal
> > attacks, I suggest you go elsewhere.
> 
> Then stick to the topic at hand, suggest positive changes, and cut the
> crap with implied personal attacks like the above. If you hadn't pull
> the discussion to that point, I wouldn't have reacted that way. It's
> completely juvenile behavior from you and you can't expect me or
> anybody else to take it sitting down.

This is also email, so misunderstanding and misinterpretations do
happen. If that's the case, then I'm sorry to misunderstand you and then
get upset, but next time be more specific about improving this code and
other things related to it.

bill

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Jens Axboe]

On Fri, Oct 22 2004, Bill Huey wrote:
> On Fri, Oct 22, 2004 at 02:06:37AM -0700, Bill Huey wrote:
> > On Fri, Oct 22, 2004 at 10:59:28AM +0200, Jens Axboe wrote:
> > > *plonk*
> > > 
> > > If you can't stand criticism without resorting to feeble personal
> > > attacks, I suggest you go elsewhere.
> > 
> > Then stick to the topic at hand, suggest positive changes, and cut the
> > crap with implied personal attacks like the above. If you hadn't pull
> > the discussion to that point, I wouldn't have reacted that way. It's
> > completely juvenile behavior from you and you can't expect me or
> > anybody else to take it sitting down.
> 
> This is also email, so misunderstanding and misinterpretations do
> happen. If that's the case, then I'm sorry to misunderstand you and then
> get upset, but next time be more specific about improving this code and
> other things related to it.

I've been as clear as I know how on the matter of semaphore use in
Linux. I've made no comments at all on improving your deadlock
detection scheme.

-- 
Jens Axboe

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Bill Huey]

On Fri, Oct 22, 2004 at 11:20:59AM +0200, Jens Axboe wrote:
> I've been as clear as I know how on the matter of semaphore use in
> Linux. I've made no comments at all on improving your deadlock
> detection scheme.

True, but "...deadlock detection breaks" is a negative comment about
the deadlock detector without a positive suggestion to change it, is
it not ? if so, then suggest a change to be made and it'll get
implementated somehow.

bill

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Jens Axboe]

On Fri, Oct 22 2004, Bill Huey wrote:
> On Fri, Oct 22, 2004 at 11:20:59AM +0200, Jens Axboe wrote:
> > I've been as clear as I know how on the matter of semaphore use in
> > Linux. I've made no comments at all on improving your deadlock
> > detection scheme.
> 
> True, but "...deadlock detection breaks" is a negative comment about
> the deadlock detector without a positive suggestion to change it, is
> it not ? if so, then suggest a change to be made and it'll get
> implementated somehow.

It's a statement about the deadlock detection which is true, it's not a
negative comment. A negative comment would be something ala "the
deadlock detection code is crap". Note, to avoid further confusion in
this thread: I have not read the deadlock detection code, nor do I
intend to. The sentence is only an example of what a negative comment
would look like, in no way does it reflect my view of the deadlock
detection code. End disclaimer.

As I said, I have no personal motivation to work on the deadlock
detection. My interest in the thread pertained only to code in the
kernel and its use of semaphores - something that we already cleared up
many mails ago.

So, please, lets just end it here. This branch of the thread has already
dragged on for way too long.

-- 
Jens Axboe

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Jens Axboe]

On Fri, Oct 22 2004, Bill Huey wrote:
> On Fri, Oct 22, 2004 at 10:59:28AM +0200, Jens Axboe wrote:
> > On Fri, Oct 22 2004, Bill Huey wrote:
> > > On Fri, Oct 22, 2004 at 08:19:01AM +0200, Jens Axboe wrote:
> > > > It has to go, why? Because your deadlock detection breaks? Doesn't seem
> > > > a very strong reason to me at all, sorry.
> > > 
> > > The deadlock detector is needed. Whether you understand that or not is
> > > irrelevant to the current work that's being done. And your idiot attacks
> > > against it doesn't correct these issues nor does it gain credibility
> > > with the audience that does find it useful.
> > 
> > *plonk*
> > 
> > If you can't stand criticism without resorting to feeble personal
> > attacks, I suggest you go elsewhere.
> 
> Then stick to the topic at hand, suggest positive changes, and cut the
> crap with implied personal attacks like the above. If you hadn't pull
> the discussion to that point, I wouldn't have reacted that way. It's
> completely juvenile behavior from you and you can't expect me or
> anybody else to take it sitting down.

What mails are you reading?!

Personally, I could not care less about the deadlock detection. If it's
a priority for you personally or due to corporate reasons, fine, but
don't involve me.

I have made no attacks on your deadlock detection other than to state
the obvious - that it has cases where it triggers on perfectly legit
code. If you read that as "implied personal attacks" or "juvenile
behaviour" then you need to grow thicker skin. The only personal attacks
here are the ones coming from you.

-- 
Jens Axboe

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Thomas Gleixner]

On Fri, 2004-10-22 at 11:06, Bill Huey wrote:
> On Fri, Oct 22, 2004 at 10:59:28AM +0200, Jens Axboe wrote:
> > On Fri, Oct 22 2004, Bill Huey wrote:
> > > On Fri, Oct 22, 2004 at 08:19:01AM +0200, Jens Axboe wrote:
> > > > It has to go, why? Because your deadlock detection breaks? Doesn't seem
> > > > a very strong reason to me at all, sorry.
> > > 
> > > The deadlock detector is needed. Whether you understand that or not is
> > > irrelevant to the current work that's being done. And your idiot attacks
> > > against it doesn't correct these issues nor does it gain credibility
> > > with the audience that does find it useful.
> > 
> > *plonk*
> > 
> > If you can't stand criticism without resorting to feeble personal
> > attacks, I suggest you go elsewhere.
> 
> Then stick to the topic at hand, suggest positive changes, and cut the
> crap with implied personal attacks like the above. If you hadn't pull
> the discussion to that point, I wouldn't have reacted that way. It's
> completely juvenile behavior from you and you can't expect me or
> anybody else to take it sitting down.

Stop that now !

You have started personal attacks.

This flame was already off. What the heck are you trying to achieve with
this ?

tglx

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

* Bill Huey <bhuey@lnxw.com> wrote:

> On Fri, Oct 22, 2004 at 08:19:01AM +0200, Jens Axboe wrote:
> > It has to go, why? Because your deadlock detection breaks? Doesn't seem
> > a very strong reason to me at all, sorry.
> 
> The deadlock detector is needed. Whether you understand that or not is
> irrelevant to the current work that's being done. And your idiot
> attacks against it doesn't correct these issues nor does it gain
> credibility with the audience that does find it useful.

oh no!

/me watches the flames fan out again as a bushfire

do you expect there to be any meaningful technical discussion resulting
out of you calling Jens' (valid) comments 'idiot attacks'? Fact is,
upstream couldnt care less about PREEMPT_REALTIME and/or the deadlock
detector at this point. Upstream _never_ cared about any not yet merged
(and in this case, completely unfinished) feature.

/me apologizes to Jens

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Christoph Hellwig]

On Fri, Oct 22, 2004 at 01:50:07AM -0700, Bill Huey wrote:
> On Fri, Oct 22, 2004 at 08:19:01AM +0200, Jens Axboe wrote:
> > It has to go, why? Because your deadlock detection breaks? Doesn't seem
> > a very strong reason to me at all, sorry.
> 
> The deadlock detector is needed. Whether you understand that or not is
> irrelevant to the current work that's being done. And your idiot attacks
> against it doesn't correct these issues nor does it gain credibility
> with the audience that does find it useful.

*plonk*

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Timothy Miller]

Bill Huey (hui) wrote:

> 
> 
> You use a semaphore to protect data, a completion isn't protecting data
> but preserving a certain kind of wait ordering in the code. The
> possibility of overloading the current mutex_t for PI makes for a conceptual
> mismatch when used in this case since having a kind of priority for
> completions is a bit odd. It's better to flat out use a completion
> instead, IMO.
> 


Could you please define "completion" for me in this context?

Thanks.

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Thomas Gleixner]

On Fri, 2004-10-22 at 00:42, Timothy Miller wrote:
> Bill Huey (hui) wrote:
> > You use a semaphore to protect data, a completion isn't protecting data
> > but preserving a certain kind of wait ordering in the code. The
> > possibility of overloading the current mutex_t for PI makes for a conceptual
> > mismatch when used in this case since having a kind of priority for
> > completions is a bit odd. It's better to flat out use a completion
> > instead, IMO.
> > 
> 
> 
> Could you please define "completion" for me in this context?

A triggers B to exit and must wait until B has exited. It waits for
completion of exit.

A triggers B to execute a command and must wait until B has done so.  It
waits for completion of the command.

Mutexes are used for that, but that's not the intended functionality of
a mutex. Of course it works as long as you do no owner checks on the
mutexes.

A {
	init_MUTEX_LOCKED(m)
	trigger B
	down(m)	<----- recursion, because A owns it already
}

The completion is designed for that and should be used IMHO. Mutexe were
used for that, because the ancestors of completion, sleep_on...(), are
racy.

tglx

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

* Rui Nuno Capela <rncbc@rncbc.org> wrote:

>    One of the signs that there's real trouble in here can be seen on
> the following complete dmesg output (which was even a miracle to be
> captured at all). This shows the complete bootstrap and init sequences
> and at the end one fatal crash while plugging an USB flash memory
> stick (usb-storage). This has been already reported earlier yesterday,
> but I just want to make it here, as the evidence-at-hand.
> 
>    After this precise occurence, the system becomes very flaky,
> unreliable and often ends up freezing to death.

for the sake of testing could you disable CONFIG_USB and see whether the
instability is truly directly related to the USB crash, as you suspect?
Such a kernel crash can often destabilize other parts of the kernel.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Rui Nuno Capela]

Ingo Molnar wrote:
>
> * Rui Nuno Capela <rncbc@rncbc.org> wrote:
>
>>    One of the signs that there's real trouble in here can be seen on
>> the following complete dmesg output (which was even a miracle to be
>> captured at all). This shows the complete bootstrap and init sequences
>> and at the end one fatal crash while plugging an USB flash memory
>> stick (usb-storage). This has been already reported earlier yesterday,
>> but I just want to make it here, as the evidence-at-hand.
>>
>>    After this precise occurence, the system becomes very flaky,
>> unreliable and often ends up freezing to death.
>
> for the sake of testing could you disable CONFIG_USB and see whether the
> instability is truly directly related to the USB crash, as you suspect?
> Such a kernel crash can often destabilize other parts of the kernel.
>

Just tested with CONFIG_USB off, and can't test the usb-storage crash, of
course. However, jackd  is still freezing to death. No console, nor syslog
output can be found. The system just dies sometime after some jack client
is launched. Will try further.

I'm on the way to test Thomas Gleixner's patch...

BBL
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Rui Nuno Capela]

[-- Attachment #1: Type: text/plain, Size: 736 bytes --]

>>
>> for the sake of testing could you disable CONFIG_USB and see whether the
>> instability is truly directly related to the USB crash, as you suspect?
>> Such a kernel crash can often destabilize other parts of the kernel.
>>
>
> Just tested with CONFIG_USB off, and can't test the usb-storage crash, of
> course. However, jackd  is still freezing to death. No console, nor syslog
> output can be found. The system just dies sometime after some jack client
> is launched. Will try further.
>
> I'm on the way to test Thomas Gleixner's patch...
>

OK. Thomas patch solves the usb-storage crash, but I added an oneliner
change to it, to make it build. Corrected patch is appended.

Thanks.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

[-- Attachment #2: realtime-preempt-2.6.9-rc4-mm1-U8.1_usb-storage.patch --]
[-- Type: text/plain, Size: 2163 bytes --]

diff -urN 2.6.9-rc4-mm1-RT-U8/drivers/usb/storage/scsiglue.c
2.6.9-rc4-mm1-RT-U8.1/drivers/usb/storage/scsiglue.c
--- 2.6.9-rc4-mm1-RT-U8/drivers/usb/storage/scsiglue.c	2004-10-12
09:41:44.000000000 +0200
+++ 2.6.9-rc4-mm1-RT-U8.1/drivers/usb/storage/scsiglue.c	2004-10-21
11:45:14.000000000 +0200
@@ -187,7 +187,7 @@
 	us->srb = srb;
 
 	/* wake up the process task */
-	up(&(us->sema));
+	complete(&(us->done));
 
 	return 0;
 }
diff -urN 2.6.9-rc4-mm1-RT-U8/drivers/usb/storage/usb.c
2.6.9-rc4-mm1-RT-U8.1/drivers/usb/storage/usb.c
--- 2.6.9-rc4-mm1-RT-U8/drivers/usb/storage/usb.c	2004-10-12
09:41:44.000000000 +0200
+++ 2.6.9-rc4-mm1-RT-U8.1/drivers/usb/storage/usb.c	2004-10-21
11:45:34.000000000 +0200
@@ -299,7 +299,7 @@
 
 	for(;;) {
 		US_DEBUGP("*** thread sleeping.\n");
-		if(down_interruptible(&us->sema))
+		if(wait_for_completion_interruptible(&us->done))
 			break;
 			
 		US_DEBUGP("*** thread awakened.\n");
@@ -845,7 +845,7 @@
 		scsi_unlock(us->host);
 		up(&us->dev_semaphore);
 
-		up(&us->sema);
+		complete(&us->done);
 		wait_for_completion(&us->notify);
 	}

@@ -941,7 +941,7 @@
 	}
 	memset(us, 0, sizeof(struct us_data));
 	init_MUTEX(&(us->dev_semaphore));
-	init_MUTEX_LOCKED(&(us->sema));
+	init_completion(&(us->done));
 	init_completion(&(us->notify));
 	init_waitqueue_head(&us->dev_reset_wait);
 	init_waitqueue_head(&us->scsi_scan_wait);
diff -urN 2.6.9-rc4-mm1-RT-U8/drivers/usb/storage/usb.h
2.6.9-rc4-mm1-RT-U8.1/drivers/usb/storage/usb.h
--- 2.6.9-rc4-mm1-RT-U8/drivers/usb/storage/usb.h	2004-10-12
09:41:44.000000000 +0200
+++ 2.6.9-rc4-mm1-RT-U8.1/drivers/usb/storage/usb.h	2004-10-21
11:45:13.000000000 +0200
@@ -159,8 +159,8 @@
 	dma_addr_t		iobuf_dma;
 
 	/* mutual exclusion and synchronization structures */
-	struct semaphore	sema;		 /* to sleep thread on   */
-	struct completion	notify;		 /* thread begin/end	 */
+	struct completion	done;   	 /* to sleep thread on   */
+	struct completion	notify; 	 /* thread begin/end	 */
 	wait_queue_head_t	dev_reset_wait;  /* wait during reset    */
 	wait_queue_head_t	scsi_scan_wait;	 /* wait before scanning */
 	struct completion	scsi_scan_done;	 /* scan thread end	 */

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Thomas Gleixner]

On Thu, 2004-10-21 at 11:12, Rui Nuno Capela wrote:

Can you try that one ?

diff -urN 2.6.9-rc4-mm1-RT-U8/drivers/usb/storage/scsiglue.c
2.6.9-rc4-mm1-RT-U8.1/drivers/usb/storage/scsiglue.c
--- 2.6.9-rc4-mm1-RT-U8/drivers/usb/storage/scsiglue.c	2004-10-12
09:41:44.000000000 +0200
+++ 2.6.9-rc4-mm1-RT-U8.1/drivers/usb/storage/scsiglue.c	2004-10-21
11:45:14.000000000 +0200
@@ -187,7 +187,7 @@
 	us->srb = srb;
 
 	/* wake up the process task */
-	up(&(us->sema));
+	complete(&(us->done));
 
 	return 0;
 }
diff -urN 2.6.9-rc4-mm1-RT-U8/drivers/usb/storage/usb.c
2.6.9-rc4-mm1-RT-U8.1/drivers/usb/storage/usb.c
--- 2.6.9-rc4-mm1-RT-U8/drivers/usb/storage/usb.c	2004-10-12
09:41:44.000000000 +0200
+++ 2.6.9-rc4-mm1-RT-U8.1/drivers/usb/storage/usb.c	2004-10-21
11:45:34.000000000 +0200
@@ -299,7 +299,7 @@
 
 	for(;;) {
 		US_DEBUGP("*** thread sleeping.\n");
-		if(down_interruptible(&us->sema))
+		if(wait_for_completion_interruptible(&us->done))
 			break;
 			
 		US_DEBUGP("*** thread awakened.\n");
@@ -941,7 +941,7 @@
 	}
 	memset(us, 0, sizeof(struct us_data));
 	init_MUTEX(&(us->dev_semaphore));
-	init_MUTEX_LOCKED(&(us->sema));
+	init_completion(&(us->done));
 	init_completion(&(us->notify));
 	init_waitqueue_head(&us->dev_reset_wait);
 	init_waitqueue_head(&us->scsi_scan_wait);
diff -urN 2.6.9-rc4-mm1-RT-U8/drivers/usb/storage/usb.h
2.6.9-rc4-mm1-RT-U8.1/drivers/usb/storage/usb.h
--- 2.6.9-rc4-mm1-RT-U8/drivers/usb/storage/usb.h	2004-10-12
09:41:44.000000000 +0200
+++ 2.6.9-rc4-mm1-RT-U8.1/drivers/usb/storage/usb.h	2004-10-21
11:45:13.000000000 +0200
@@ -159,8 +159,8 @@
 	dma_addr_t		iobuf_dma;
 
 	/* mutual exclusion and synchronization structures */
-	struct semaphore	sema;		 /* to sleep thread on   */
-	struct completion	notify;		 /* thread begin/end	 */
+	struct completion	done;   	 /* to sleep thread on   */
+	struct completion	notify; 	 /* thread begin/end	 */
 	wait_queue_head_t	dev_reset_wait;  /* wait during reset    */
 	wait_queue_head_t	scsi_scan_wait;	 /* wait before scanning */
 	struct completion	scsi_scan_done;	 /* scan thread end	 */

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Rui Nuno Capela]

Rui Nuno Capela wrote:
> Ingo Molnar wrote:
>>
>> i have released the -U8 Real-Time Preemption patch:
>>
>>   http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U8
>>
> [...]
>
> b) Laptop P4 2.533Ghz UP (Mandrake 10.1c)
>    config-2.6.9-rc4-mm1-RT-U8.1.gz
>
>    This box was known to work without major issues until U4. With U8 it's
> a real pain. Once trivial operations turns out fatal now. Running jackd
> -R, which has been a flagship before, now freezes the whole system in
> no time. (I'll take some netconsole capture sessions later)
>

OK.

Now that the usb-storage crash has been ironed out, thanks to Thomas
Gleixner's patch, I proceeded with some local experiments regarding the
jackd -R issue.

The fact is jackd -R (realtime mode; SCHED_FIFO) hosing the system, and
thats exposed as soon as some jack audio client application enters into
the chain.

Running jackd non-realtime (SCHED_OTHER) does not expose this problem, so
I think it's a scheduling related one.

With a default scenario, with all IRQ handlers under SCHED_OTHER
scheduling class and default priority, running jackd -R freezes completely
the system. Only a hard-reset or power-off is the way out.

Then I try tweaking the keyboard (IRQs 1,12), rtc (IRQ 8) and soundcard
(IRQ 5) scheduling policies to SCHED_FIFO and priority to something higher
than jackd's (e.g. chrt --fifo 60).

This way, running jack -R still hoses the system, in a somewhat less
egoistic manner, but still seems that it's the only process running on the
system taking full control of it. The evidence I could find was that
jackd's verbose output keeps pumping, as it would usually, but all the
rest poor things freeze to death. This time however, magic SysRq is of
some use, barely thanks to the i8042 IRQ scheduling promotion.

So it all seems that jackd -R is not crashing, nor anything else, for this
matter.

I really hate to say this, but this should be investigated for the RT
patch sake, obviously because the only purpose I find to it is precisely
running jackd -R, and I can swear it has been near perfection until U4
inclusive (think it was called VP back then :).

I hope I'm not the only one.

(IIRC Florian Schmidt was experiencing something similar, like system
intermitence, pauses, whatever, while also running jackd -R.)

Strange enough, all this is running on another SMP/HT box of mine, without
major issues. Guess that SMP makes the difference here.

But wait, now I remember to notice something there yet: running some jack
clients (e.g. fluidsynth) is much more expensive than usual wrt.CPU usage,
reaching very unusual levels above 40% sustained (this is actual CPU% as
reported by procps tools, not the DSP usage as reported by jack itself).
IMHO the SMP/HT effect just seems to mask the real trouble, as the
increased cost affects only one of the (virtual) CPUs.

I wonder if this does ring some bell out there. :)

Cheers,
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

* Rui Nuno Capela <rncbc@rncbc.org> wrote:

> The fact is jackd -R (realtime mode; SCHED_FIFO) hosing the system,
> and thats exposed as soon as some jack audio client application enters
> into the chain.
> 
> Running jackd non-realtime (SCHED_OTHER) does not expose this problem,
> so I think it's a scheduling related one.

i tried to pole jackd a little bit (just using things like
jack_freewheel and jack_impulse_grabber - i dont even know what they 
do), and got jackd into some sort of userspace loop:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 2558 root      16   0 27900 1852 2152 S 97.8  0.8   2:36.38 jackd

attaching gdb to it shows:

 Loaded symbols for /usr/local/lib/jack/jack_oss.so
 0xffffe410 in ?? ()
 (gdb) bt
 #0  0xffffe410 in ?? ()
 #1  0xbffff7f8 in ?? ()
 #2  0x00000a67 in ?? ()
 #3  0x00000000 in ?? ()
 #4  0x4db8adf8 in pthread_join () from /lib/tls/libpthread.so.0
 #5  0xb77d6e66 in oss_driver_stop (driver=0x8055938) at  oss_driver.c:696
 #6  0x0804ba03 in jack_engine_delete (engine=0x805c308) at  engine.c:2466
 #7  0x0804ade7 in main (argc=3, argv=0xbffffb44) at jackd.c:207
 (gdb)

it's looping somewhere in the pthread code, and it does no system-calls
at all and thus no scheduling as well.

i dont know much about jackit, and it could easily be that something in
this kernel broke its interaction with pthread, but it seems to me that
this loop is in userspace and is only 'fatal' if the looping thread runs
at SCHED_FIFO priority. Could someone with more jackit experience try to
figure out what's going on here?

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> i tried to pole jackd a little bit (just using things like
> jack_freewheel and jack_impulse_grabber - i dont even know what they 
> do), and got jackd into some sort of userspace loop:
> 
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>  2558 root      16   0 27900 1852 2152 S 97.8  0.8   2:36.38 jackd

ah ... i should have guessed that "jack_freewheel y" puts jackd into ...
freewheeling mode. So this is by design.

i still suspect that it's some sort of userspace loop causing the jackd
problems - just that under SCHED_OTHER you dont normally notice it,
while with jack -R it's fatal.

	Ingo

Re: [patch] Real-Time Preemption, -RT-2.6.9-rc4-mm1-U8

[Rui Nuno Capela]

> Rui Nuno Capela wrote:
>> Ingo Molnar wrote:
>>>
>>> i have released the -U8 Real-Time Preemption patch:
>>>
>>>   http://redhat.com/~mingo/realtime-preempt/realtime-preempt-2.6.9-rc4-mm1-U8
>>>
> [...]
>
> The fact is jackd -R (realtime mode; SCHED_FIFO) hosing the system, and
> thats exposed as soon as some jack audio client application enters into
> the chain.
>
> Running jackd non-realtime (SCHED_OTHER) does not expose this problem, so
> I think it's a scheduling related one.
>
> [...]


After some trial-and-error cycle, changing kernel configuration options, I
come to believe the obvious, that this jackd -R nasty behavior seems to be
present (only) if PREEMPT_REALTIME is set (Y).

When PREEMPT_REALTIME is not set (N), it just runs and I can throw any
client at 'jackd -R' without hosing the whole system. However, I'm seeing
plenty of these:

BUG: scheduling while atomic: jackd/0x00000002/3968
caller is schedule_timeout+0x5a/0xa8
 [<c0104ec8>] dump_stack+0x1e/0x20 (20)
 [<c02cd6d2>] __schedule+0x4c4/0x69e (76)
 [<c02ce45f>] schedule_timeout+0x5a/0xa8 (60)
 [<c0169c75>] do_poll+0x9b/0xb9 (48)
 [<c0169e02>] sys_poll+0x16f/0x225 (76)
 [<c010408d>] sysenter_past_esp+0x52/0x71 (-8124)
preempt count: 00000003
. 3-level deep critical section nesting:
.. entry 1: ipc_lock_writer+0x2f/0xae / (sys_shmctl+0x88/0x895)
.. entry 2: ipc_lock_writer+0xa3/0xae / (sys_shmctl+0x88/0x895)
.. entry 3: print_traces+0x16/0x4a / (dump_stack+0x1e/0x20)

BUG: sleeping function called from invalid context jackd(3968) at
mm/slab.c:2055
in_atomic():1 [00000002], irqs_disabled():0
 [<c0104ec8>] dump_stack+0x1e/0x20 (20)
 [<c011505f>] __might_sleep+0xb2/0xc5 (36)
 [<c013e7f8>] __kmalloc+0xa3/0xaa (28)
 [<c0169d60>] sys_poll+0xcd/0x225 (76)
 [<c010408d>] sysenter_past_esp+0x52/0x71 (-8124)
preempt count: 00000003
. 3-level deep critical section nesting:
.. entry 1: ipc_lock_writer+0x2f/0xae / (sys_shmctl+0x88/0x895)
.. entry 2: ipc_lock_writer+0xa3/0xae / (sys_shmctl+0x88/0x895)
.. entry 3: print_traces+0x16/0x4a / (dump_stack+0x1e/0x20)



OTOH, I do get in some trouble elsewhere, but not related to jackd. For
example, the system hangs on udev, never managed to shutdown cleanly, and
some system monitoring applications just keeps failing completely (e.g.
gkrellm).

But I found this on late init:

BUG: Unable to handle kernel NULL pointer dereference at virtual address
00000000
 printing eip:
c01cbb7d
*pde = 00000000
Oops: 0000 [#1]
PREEMPT
Modules linked in: realtime commoncap snd_seq_oss snd_seq_midi_event
snd_seq snd_pcm_oss snd_mixer_oss snd_usb_usx2y snd_usb_lib snd_rawmidi
snd_seq_device snd_hwdep snd_ali5451 snd_ac97_codec snd_pcm snd_timer
snd_page_alloc snd soundcore prism2_cs p80211 ds yenta_socket pcmcia_core
natsemi crc32 loop subfs evdev ohci_hcd usbcore thermal processor fan
button battery ac
CPU:    0
EIP:    0060:[<c01cbb7d>]    Not tainted VLI
EFLAGS: 00210246   (2.6.9-rc4-mm1-U9.2)
EIP is at acpi_os_signal_semaphore+0x30/0x4e
eax: df75b700   ebx: 00000001   ecx: 00000000   edx: 00000010
esi: c155c400   edi: 00000000   ebp: de48dd9c   esp: de48dd98
ds: 007b   es: 007b   ss: 0068   preempt: 00000001
Process kdeinit (pid: 3862, threadinfo=de48c000 task=de5af400)
Stack: df750700 de48ddac c01d4b88 df74b160 00000001 de48ddc4 c01d67a2
df750700
       df750700 c155c400 c155c400 de48ddd8 c01d3abe df750700 c155c400
00000000
       de48ddf8 c01cd2e2 c155c400 00000000 c149c820 c155c400 c155c5ec
00000000
Call Trace:
 [<c0104e94>] show_stack+0x80/0x96 (28)
 [<c010502f>] show_registers+0x165/0x1de (56)
 [<c0105241>] die+0xf6/0x191 (64)
 [<c01123ab>] do_page_fault+0x483/0x6a4 (212)
 [<c0104af1>] error_code+0x2d/0x38 (72)
 [<c01d4b88>] acpi_ex_system_release_mutex+0x28/0x2a (16)
 [<c01d67a2>] acpi_ex_release_mutex+0x135/0x154 (24)
 [<c01d3abe>] acpi_ex_opcode_1A_0T_0R+0x2a/0x92 (20)
 [<c01cd2e2>] acpi_ds_exec_end_op+0xb4/0x28e (32)
 [<c01db23e>] acpi_ps_parse_loop+0x528/0x810 (40)
 [<c01db57d>] acpi_ps_parse_aml+0x57/0x1c2 (32)
 [<c01dbea5>] acpi_psx_execute+0x15d/0x1c4 (28)
 [<c01d914d>] acpi_ns_execute_control_method+0x41/0x51 (20)
 [<c01d90f2>] acpi_ns_evaluate_by_handle+0x74/0x8e (16)
 [<c01d8fed>] acpi_ns_evaluate_relative+0xa9/0xc5 (32)
 [<c01d88a5>] acpi_evaluate_object+0xfd/0x1ae (52)
 [<c01cbedd>] acpi_evaluate_integer+0x32/0x4f (52)
 [<e001a06c>] acpi_button_state_seq_show+0x27/0x64 [button] (32)
 [<c0175562>] seq_read+0xd3/0x2cf (60)
 [<c015461c>] vfs_read+0xc1/0x13a (44)
 [<c015490b>] sys_read+0x4b/0x75 (44)
 [<c010408d>] sysenter_past_esp+0x52/0x71 (-8124)
preempt count: 00000002
. 2-level deep critical section nesting:
.. entry 1: die+0x3a/0x191 / (do_page_fault+0x483/0x6a4)
.. entry 2: print_traces+0x16/0x4a / (show_stack+0x80/0x96)

Code: 4d 08 8b 5d 0c 85 c9 0f 94 c2 85 db 0f 94 c0 09 d0 ba 01 10 00 00 a8
01 75 28 83 fb 01 66 ba 10 00 77 1f ff 01 0f 8e d2 00 00 00 <8b> 01 48 7e
10 68 29 7e 2e c0 e8 5a c3 f4 ff 5b e8 18 93 f3 ff


As it seemed an ACPI issue, I turned it off and the troubles went away.
Again, all this is happening with PREEMPT_REALTIME off.

Bye now.
-- 
rncbc aka Rui Nuno Capela
rncbc@rncbc.org