queue_work from interrupt Real time preemption2.6.11-rc2-RT-V0.7.37-03

queue_work from interrupt Real time preemption2.6.11-rc2-RT-V0.7.37-03

[Mark Gross]

I'm working on a tweak to the preepmtive soft IRQ implementation using work 
queues and I'm having problems with a BUG assert when trying to queue_work.

Souldn't I be able to call queue_work form ISR context?

--mgross 

---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c0140f5d>] .... print_traces+0x1d/0x60
.....[<c01042a3>] ..   ( <= dump_stack+0x23/0x30)

BUG: sleeping function called from invalid context IRQ 20(2039) at kernel/rt.c
in_atomic():0 [00000000], irqs_disabled():1
 [<c01042a3>] dump_stack+0x23/0x30 (20)
 [<c011f315>] __might_sleep+0xe5/0x100 (36)
 [<c013cd19>] __spin_lock+0x39/0x60 (24)
 [<c013cded>] _spin_lock_irqsave+0x1d/0x30 (16)
 [<c0134b32>] __queue_work+0x22/0x70 (28)
 [<c0134bd8>] queue_work+0x58/0xa0 (32)
 [<c01287cd>] __raise_softirq+0x2d/0x50 (12)
 [<f898b944>] e100_intr+0x124/0x170 [e100] (36)
 [<c0147ab9>] handle_IRQ_event+0x69/0xf0 (52)
 [<c0148365>] do_hardirq+0xa5/0x110 (40)
 [<c01484f1>] do_irqd+0x121/0x1f0 (48)
 [<c013990b>] kthread+0xbb/0xc0 (48)
 [<c01013f9>] kernel_thread_helper+0x5/0xc (154181652)
---------------------------

| preempt count: 00010001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c0140f5d>] .... print_traces+0x1d/0x60
.....[<c01042a3>] ..   ( <= dump_stack+0x23/0x30)

BUG: sleeping function called from invalid context swapper(0) at kernel/rt.c:1
in_atomic():1 [00010000], irqs_disabled():1
 [<c01042a3>] dump_stack+0x23/0x30 (20)
 [<c011f315>] __might_sleep+0xe5/0x100 (36)
 [<c013cd19>] __spin_lock+0x39/0x60 (24)
 [<c013cded>] _spin_lock_irqsave+0x1d/0x30 (16)
 [<c0134b32>] __queue_work+0x22/0x70 (28)
 [<c0134bd8>] queue_work+0x58/0xa0 (32)
 [<c0128aee>] raise_softirq+0x3e/0xb0 (20)
 [<c012d395>] update_process_times+0x45/0x110 (32)
 [<c01153ba>] smp_apic_timer_interrupt+0xea/0xf0 (36)
 [<c0103e18>] apic_timer_interrupt+0x1c/0x24 (60)
 [<c0101163>] cpu_idle+0x83/0x90 (24)
 [<c0404a8d>] start_kernel+0x19d/0x1e0 (32)
 [<c010020e>] 0xc010020e (1077727247)
---------------------------

Re: queue_work from interrupt Real time preemption2.6.11-rc2-RT-V0.7.37-03

[Steven Rostedt]

On Mon, 2005-02-14 at 12:40 -0800, Mark Gross wrote:
> I'm working on a tweak to the preepmtive soft IRQ implementation using work 
> queues and I'm having problems with a BUG assert when trying to queue_work.
> 
> Souldn't I be able to call queue_work form ISR context?

Yes, but not with interrupts disabled.
> 
> --mgross 
> 
> ---------------------------
> | preempt count: 00000001 ]
> | 1-level deep critical section nesting:
> ----------------------------------------
> .. [<c0140f5d>] .... print_traces+0x1d/0x60
> .....[<c01042a3>] ..   ( <= dump_stack+0x23/0x30)
> 
> BUG: sleeping function called from invalid context IRQ 20(2039) at kernel/rt.c
> in_atomic():0 [00000000], irqs_disabled():1

Here you have interrupts disabled. Since you are tweaking the softirq I
don't know your code, but the kernel should not schedule after turning
off interrupts, and the spinlocks under the PREEMPT kernel, may now
sleep (unless they are raw_spin_locks).  Here we also see that
queue_work calls spin_lock_irqsave.  I'm suspecting that you turned off
interrupts somewhere.

-- Steve

Re: queue_work from interrupt Real time preemption2.6.11-rc2-RT-V0.7.37-03

[Mark Gross]

On Monday 14 February 2005 13:24, Steven Rostedt wrote:
> On Mon, 2005-02-14 at 12:40 -0800, Mark Gross wrote:
> > I'm working on a tweak to the preepmtive soft IRQ implementation using
> > work queues and I'm having problems with a BUG assert when trying to
> > queue_work.
> >
> > Souldn't I be able to call queue_work form ISR context?
>
> Yes, but not with interrupts disabled.
>

Hmm.  It seems to me that one should be able to call queue_work from wherever 
you can call raise_softirq.  This constraint adds a bit of asymetry in the 
deffered processing API's


> > --mgross
> >
> > ---------------------------
> >
> > | preempt count: 00000001 ]
> > | 1-level deep critical section nesting:
> >
> > ----------------------------------------
> > .. [<c0140f5d>] .... print_traces+0x1d/0x60
> > .....[<c01042a3>] ..   ( <= dump_stack+0x23/0x30)
> >
> > BUG: sleeping function called from invalid context IRQ 20(2039) at
> > kernel/rt.c in_atomic():0 [00000000], irqs_disabled():1
>
> Here you have interrupts disabled. Since you are tweaking the softirq I
> don't know your code, but the kernel should not schedule after turning
> off interrupts, and the spinlocks under the PREEMPT kernel, may now
> sleep (unless they are raw_spin_locks).  Here we also see that
> queue_work calls spin_lock_irqsave.  I'm suspecting that you turned off
> interrupts somewhere.
>
> -- Steve

I'll post my code soon, I hope.  I now need to work around this API problem :(

thanks, 

--mgross

Re: queue_work from interrupt Real time preemption2.6.11-rc2-RT-V0.7.37-03

[Ingo Molnar]

* Mark Gross <mgross@linux.intel.com> wrote:

> On Monday 14 February 2005 13:24, Steven Rostedt wrote:
> > On Mon, 2005-02-14 at 12:40 -0800, Mark Gross wrote:
> > > I'm working on a tweak to the preepmtive soft IRQ implementation using
> > > work queues and I'm having problems with a BUG assert when trying to
> > > queue_work.
> > >
> > > Souldn't I be able to call queue_work form ISR context?
> >
> > Yes, but not with interrupts disabled.
> 
> Hmm.  It seems to me that one should be able to call queue_work from
> wherever you can call raise_softirq.  This constraint adds a bit of
> asymetry in the deffered processing API's

one solution is to use the local_irq_*_nort() API variants - but it all
depends on why you had to disable interrupts.

Almost always irq-disabling done in conjunction with spinlocks, and the
spin_lock_irq*() variants do not disable interrupts on PREEMPT_RT. I
kept the assymetry of the local_irq*() APIs because in most cases they
are used directly interrupts need to be disabled.

it is also the more conservative approach, since we'll get messages like
the ones you got when it's unsafe to do it - while if local_irq_*() APIs
didnt disable interrupts we'd never know about the cases when they
_must_ be disabled.)

but yes, there's some API assymetry - which mostly comes from the fact
alone that 99.999% of the kernel is now preemptible. There's just so
much we can do to pretend that this is good'old Linux kernel semantics
:-)

	Ingo

Re: queue_work from interrupt Real time preemption2.6.11-rc2-RT-V0.7.37-03

[Mark Gross]

On Tuesday 15 February 2005 02:41, Ingo Molnar wrote:
> * Mark Gross <mgross@linux.intel.com> wrote:
> > On Monday 14 February 2005 13:24, Steven Rostedt wrote:
> > > On Mon, 2005-02-14 at 12:40 -0800, Mark Gross wrote:
> > > > I'm working on a tweak to the preepmtive soft IRQ implementation
> > > > using work queues and I'm having problems with a BUG assert when
> > > > trying to queue_work.
> > > >
> > > > Souldn't I be able to call queue_work form ISR context?
> > >
> > > Yes, but not with interrupts disabled.
> >
> > Hmm.  It seems to me that one should be able to call queue_work from
> > wherever you can call raise_softirq.  This constraint adds a bit of
> > asymetry in the deffered processing API's
>
> one solution is to use the local_irq_*_nort() API variants - but it all
> depends on why you had to disable interrupts.

I'm attempting to change the softIRQ preemption implementation to use work 
queues (one per softIRQ), that allow for runtime priority changes on a 
per-soft IRQ bases.  To do this I was trying to have raise_softirq call 
queu_work directly.  queue_work, doesn't use the *_nort() api's.

My alternitive is to put the call to queue_work into do_softIRQ.  Which seems 
to  work, but feels like a bit too much indirection to queue up the soft IRQ 
bottom half processing.


>
> Almost always irq-disabling done in conjunction with spinlocks, and the
> spin_lock_irq*() variants do not disable interrupts on PREEMPT_RT. I
> kept the assymetry of the local_irq*() APIs because in most cases they
> are used directly interrupts need to be disabled.
>
> it is also the more conservative approach, since we'll get messages like
> the ones you got when it's unsafe to do it - while if local_irq_*() APIs
> didnt disable interrupts we'd never know about the cases when they
> _must_ be disabled.)
>
> but yes, there's some API assymetry - which mostly comes from the fact
> alone that 99.999% of the kernel is now preemptible. There's just so
> much we can do to pretend that this is good'old Linux kernel semantics
>
> :-)
>
> 	Ingo

Re: queue_work from interrupt Real time preemption2.6.11-rc2-RT-V0.7.37-03

[Ingo Molnar]

* Mark Gross <mgross@linux.intel.com> wrote:

> I'm attempting to change the softIRQ preemption implementation to use
> work queues (one per softIRQ), that allow for runtime priority changes
> on a per-soft IRQ bases.  To do this I was trying to have
> raise_softirq call queu_work directly.  queue_work, doesn't use the
> *_nort() api's.
> 
> My alternitive is to put the call to queue_work into do_softIRQ. 
> Which seems to work, but feels like a bit too much indirection to
> queue up the soft IRQ bottom half processing.

correct, that may be too much of an indirection.

we cannot turn queue_work()'s spinlocks into raw spinlocks, because that
would necessiate waitqueue locks to be raw spinlocks too, which would be
generally bad. Check out how entangled the cwq->work_done and
cwq->more_work waitqueues are with cwq->lock.

also, i'm not sure about how correctly this maps to the upstream softirq
semantics. Right now softirqs are processed by a single context (per
CPU), and each softirq type is processed serially. Softirqs are
preemptable, but they dont preempt each other.  Maybe the networking
stack would break if we allowed the TIMER softirq (thread) to preempt
the NET softirq (threads) (and vice versa)?

but, prioritizing individual softirqs (within the PREEMPT_RT framework)
is something that needs to be thought about - e.g.  Mark H. Johnson
reported various PREEMPT_RT (and softirq threading) SMP artifacts that
seem to be related to the current 'all softirqs are processed by a
single thread' approach. (check Mark's postings for 'ping' latency
anomalies)

	Ingo

Re: queue_work from interrupt Real time preemption2.6.11-rc2-RT-V0.7.37-03

[David S. Miller]

On Wed, 16 Feb 2005 06:16:45 +0100
Ingo Molnar <mingo@elte.hu> wrote:


> Maybe the networking
> stack would break if we allowed the TIMER softirq (thread) to preempt
> the NET softirq (threads) (and vice versa)?

The major assumption is that softirq's run indivisibly per-cpu.
Otherwise the per-cpu queues of RX and TX packet work would
get corrupted.

See net/core/dev.c:softnet_data

Re: queue_work from interrupt Real time preemption2.6.11-rc2-RT-V0.7.37-03

[George Anzinger]

David S. Miller wrote:
> On Wed, 16 Feb 2005 06:16:45 +0100
> Ingo Molnar <mingo@elte.hu> wrote:
> 
> 
> 
>>Maybe the networking
>>stack would break if we allowed the TIMER softirq (thread) to preempt
>>the NET softirq (threads) (and vice versa)?
> 
> 
> The major assumption is that softirq's run indivisibly per-cpu.
> Otherwise the per-cpu queues of RX and TX packet work would
> get corrupted.

For what its worth, I, a short while ago, put together a workqueue package to a) 
allow easy priority setting for work queues and b) change either softirq, 
tasklet or bh code to use workqueues.  This was done mostly with CPP macros and 
a few conversion routines.  I then converted the network code to use this 
package simply by adding a key include to a couple of files.  The result worked 
on UP but ended up hanging the network code on SMP.  Everything else still 
worked, but not the net stuff.  I never ran down the problem as the "boss" was 
not interested in SMP...

George

-- 
George Anzinger   george@mvista.com
High-res-timers:  http://sourceforge.net/projects/high-res-timers/

Re: queue_work from interrupt Real time preemption2.6.11-rc2-RT-V0.7.37-03

[Mark Gross]

On Wednesday 16 February 2005 09:59, George Anzinger wrote:
> David S. Miller wrote:
> > On Wed, 16 Feb 2005 06:16:45 +0100
> >
> > Ingo Molnar <mingo@elte.hu> wrote:
> >>Maybe the networking
> >>stack would break if we allowed the TIMER softirq (thread) to preempt
> >>the NET softirq (threads) (and vice versa)?
> >
> > The major assumption is that softirq's run indivisibly per-cpu.
> > Otherwise the per-cpu queues of RX and TX packet work would
> > get corrupted.

That's a problem (for my idea).

>
> For what its worth, I, a short while ago, put together a workqueue package
> to a) allow easy priority setting for work queues and b) change either
> softirq, tasklet or bh code to use workqueues.  This was done mostly with
> CPP macros and a few conversion routines.  I then converted the network
> code to use this package simply by adding a key include to a couple of
> files.  The result worked on UP but ended up hanging the network code on
> SMP.  Everything else still worked, but not the net stuff.  I never ran
> down the problem as the "boss" was not interested in SMP...
>

Thanks,  my implementation doesn't lock up with my unit testing (scp kernel 
tarballs).  However; I did have my scheduling pollocy for the net_tx net_rx 
and timer set to SCHED_RR using the same priority for each.  I'll fiddle with 
t relative prioites across the different soft IRQ's an see how much smoke I 
can cause.

--mgross

Re: queue_work from interrupt Real time preemption2.6.11-rc2-RT-V0.7.37-03

[Steven Rostedt]

On Wed, 16 Feb 2005, David S. Miller wrote:
> On Wed, 16 Feb 2005 06:16:45 +0100
> Ingo Molnar <mingo@elte.hu> wrote:
> > Maybe the networking
> > stack would break if we allowed the TIMER softirq (thread) to preempt
> > the NET softirq (threads) (and vice versa)?
> The major assumption is that softirq's run indivisibly per-cpu.
> Otherwise the per-cpu queues of RX and TX packet work would
> get corrupted.
>
> See net/core/dev.c:softnet_data
>

How about a design to put softirq's into domains. That way you can group
together the threads that should not preempt each other (ie TX and RX) but
still allow prioritizing of different softirq threads. That way you don't
have one softirq thread preempting another thread on the same cpu that are
in the same group. I'm sure that the TX and RX is more of the exception
than the rule.

I guess you can have one implementation of this by having one thread per
group.  Of course there would have to be an API developed to group
tasklets. Also it would be a little trickier to keep individual tasklets
within a group from running on two CPUs at once, and not sacrifice TX and
RX from running together on two different CPUS. But this is all feasible.

-- Steve

Re: queue_work from interrupt Real time preemption2.6.11-rc2-RT-V0.7.37-03

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> > See net/core/dev.c:softnet_data
> 
> How about a design to put softirq's into domains. [...]

just to make sure that the context of this discussion is not lost to
David and other readers of lkml. We are not redesigning softirqs in any
way, shape or form for the normal kernel - there they remain what they
are.

This discussion is about seemless (automatic) extensions/modifications
to the softirq concept on PREEMPT_RT, for latency reduction purposes.
PREEMPT_SOFTIRQS is already such an extension.

	Ingo

Re: queue_work from interrupt Real time preemption2.6.11-rc2-RT-V0.7.37-03

[Steven Rostedt]

Damn! I'm doing this from out of town and my pine setup had a reply to to
another email account, and I didn't read this before I sent my previous
response (so Please ignore it!)

On Thu, 17 Feb 2005, Ingo Molnar wrote:
> * Steven Rostedt <rostedt@goodmis.org> wrote:
>
> > > See net/core/dev.c:softnet_data
> >
> > How about a design to put softirq's into domains. [...]
>
> just to make sure that the context of this discussion is not lost to
> David and other readers of lkml. We are not redesigning softirqs in any
> way, shape or form for the normal kernel - there they remain what they
> are.
>
> This discussion is about seemless (automatic) extensions/modifications
> to the softirq concept on PREEMPT_RT, for latency reduction purposes.
> PREEMPT_SOFTIRQS is already such an extension.
>

I'm only working on your PREEMPT_RT extension, so I wasn't thinking about
the mainline kernel.

But I'll ask again from this context. What is the plan for softirqs on the
PREEMPT_RT kernel? Are you going to thread them? Otherwise, what other way
can you preempt different softirqs?

I understand that the design of softirqs will not change for the mainline
kernel, but what changes are going to be made wrt PREEMPT_RT? If they are
going to be threaded, then grouping them would not be too much of a
problem with simple #ifdefs around the code and keep the mainline
untouched.

I may be just confused, so please enlighten me :-)

Thanks,

-- Steve

Re: queue_work from interrupt Real time preemption2.6.11-rc2-RT-V0.7.37-03

[Ingo Molnar]

* David S. Miller <davem@davemloft.net> wrote:

> > Maybe the networking
> > stack would break if we allowed the TIMER softirq (thread) to preempt
> > the NET softirq (threads) (and vice versa)?
> 
> The major assumption is that softirq's run indivisibly per-cpu.
> Otherwise the per-cpu queues of RX and TX packet work would get
> corrupted.

as long as it stays on a single CPU, could we allow softirq contexts to
preempt each other? I.e. we'd keep the per-CPU assumption (that is fair
and needed for performance anyway), but we'd allow NET_TX to preempt
NET_RX and vice versa. Would this corrupt the rx/tx queues? (i suspect
it would.)

(anyway, by adding an explicit no-preempt section around the 'take
current rx queue private, then process it' on PREEMPT_RT it could be
made safe. I'm wondering whether there are any other deeper assumptions
about atomic separation of softirq contexts.)

	Ingo

Re: queue_work from interrupt Real time preemption2.6.11-rc2-RT-V0.7.37-03

[Steven Rostedt]

On Thu, 17 Feb 2005, Ingo Molnar wrote:
> as long as it stays on a single CPU, could we allow softirq contexts to
> preempt each other? I.e. we'd keep the per-CPU assumption (that is fair
> and needed for performance anyway), but we'd allow NET_TX to preempt
> NET_RX and vice versa. Would this corrupt the rx/tx queues? (i suspect
> it would.)
>
> (anyway, by adding an explicit no-preempt section around the 'take
> current rx queue private, then process it' on PREEMPT_RT it could be
> made safe. I'm wondering whether there are any other deeper assumptions
> about atomic separation of softirq contexts.)
>

Ingo,

Wouldn't this cause a longer latency in these sections. I understand
that turning preemption off doesn't stop interrupts that are not
threaded, but especially on a UP, this would cause higher latencies for
high priority processes when a lower priority process is heavy on network
traffic.

As I mentioned earlier, what would it take to be able to group
softirq threads that should not preempt each other, but still keep
preemption available for other threads?

Actually, I guess I need to ask, what do you intend on doing to prioritize
the softirq?  Are you going to make a separate thread for each tasklet?
Once I see what you're doing, I'll make something up to help handle this
problem.

-- Steve

Re: queue_work from interrupt Real time preemption2.6.11-rc2-RT-V0.7.37-03

[Mark Gross]

On Thursday 17 February 2005 06:57, Steven Rostedt wrote:
> On Thu, 17 Feb 2005, Ingo Molnar wrote:
> > as long as it stays on a single CPU, could we allow softirq contexts to
> > preempt each other? I.e. we'd keep the per-CPU assumption (that is fair
> > and needed for performance anyway), but we'd allow NET_TX to preempt
> > NET_RX and vice versa. Would this corrupt the rx/tx queues? (i suspect
> > it would.)
> >
> > (anyway, by adding an explicit no-preempt section around the 'take
> > current rx queue private, then process it' on PREEMPT_RT it could be
> > made safe. I'm wondering whether there are any other deeper assumptions
> > about atomic separation of softirq contexts.)
>
> Ingo,
>
> Wouldn't this cause a longer latency in these sections. I understand
> that turning preemption off doesn't stop interrupts that are not
> threaded, but especially on a UP, this would cause higher latencies for
> high priority processes when a lower priority process is heavy on network
> traffic.
>
> As I mentioned earlier, what would it take to be able to group
> softirq threads that should not preempt each other, but still keep
> preemption available for other threads?

It would only take the creationt of multiple softIRQd threads per CPU.  Just 
keep net_rx and net_tx in the same work queue.


>
> Actually, I guess I need to ask, what do you intend on doing to prioritize
> the softirq?  Are you going to make a separate thread for each tasklet?
> Once I see what you're doing, I'll make something up to help handle this
> problem.
>
> -- Steve

Re: queue_work from interrupt Real time preemption2.6.11-rc2-RT-V0.7.37-03

[Ingo Molnar]

* Mark Gross <mgross@linux.intel.com> wrote:

> > As I mentioned earlier, what would it take to be able to group
> > softirq threads that should not preempt each other, but still keep
> > preemption available for other threads?
> 
> It would only take the creationt of multiple softIRQd threads per CPU.  
> Just keep net_rx and net_tx in the same work queue.

we could work around the net_rx/net_tx assumptions by moving them to the 
same softirq thread - but i'm a bit uneasy about the whole concept: e.g.  
how about SCSI softirq processing and timer softirq processing, can they 
preempt each other?

	Ingo

Re: queue_work from interrupt Real time preemption2.6.11-rc2-RT-V0.7.37-03

[Mark Gross]

On Tuesday 29 March 2005 00:57, Ingo Molnar wrote:
> * Mark Gross <mgross@linux.intel.com> wrote:
> > > As I mentioned earlier, what would it take to be able to group
> > > softirq threads that should not preempt each other, but still keep
> > > preemption available for other threads?
> >
> > It would only take the creationt of multiple softIRQd threads per CPU.
> > Just keep net_rx and net_tx in the same work queue.
>
> we could work around the net_rx/net_tx assumptions by moving them to the
> same softirq thread - but i'm a bit uneasy about the whole concept: e.g.
> how about SCSI softirq processing and timer softirq processing, can they
> preempt each other?
>
I think they can.  

BTW:

My work on this has been mostly in the context of a 2.6 kernel based 
generalization of a softIRQ as thread patch for 2.4 that enables priority 
tuning of the bottom half processing as well as /proc support for turning on 
and off the feature.   We got it to work.

However; I don't know what good workloads and metrics to measure the goodness 
of the work look like.  If folks think priority tuning of bottom half 
processing is worth persuing and can help me quantify its effectiveness 
better than running a jitter test while doing a BONNIE test run on a SCSI 
JBOD, then I'm happy to do more with this.

--mgross

Re: queue_work from interrupt Real time preemption2.6.11-rc2-RT-V0.7.37-03

[Ingo Molnar]

* Mark Gross <mgross@linux.intel.com> wrote:

> BTW:
> 
> My work on this has been mostly in the context of a 2.6 kernel based 
> generalization of a softIRQ as thread patch for 2.4 that enables 
> priority tuning of the bottom half processing as well as /proc support 
> for turning on and off the feature.  We got it to work.
> 
> However; I don't know what good workloads and metrics to measure the 
> goodness of the work look like.  If folks think priority tuning of 
> bottom half processing is worth persuing and can help me quantify its 
> effectiveness better than running a jitter test while doing a BONNIE 
> test run on a SCSI JBOD, then I'm happy to do more with this.

anything that generates a consistent interrupt rate is pretty good for 
testing. Networking is the most softirq-dependent code, so i'd say 
tbench over a real network ought to be a good benchmark.

	Ingo