IRQ handler under load - slow response

IRQ handler under load - slow response

[Arno Steffen]

Hello, maybe someone of you can give me hint what is wrong with my system.
I do have coded an irq-handler, handling changes at GPIO pins. The
response time (from changing the level at Input until output is set in
IRQ handler) is ok (<100?s)
Then I started with fork another process, which is under heavy load
(consumes a lot of CPU power). From now the response time was quite
jittering and edge changes within 10ms are handled not reliable, so it
misses some input changes.
It seems that IRQ is not responding so fast, probably because of some
kind of process switch???

In application I announce the IRQ with
    sigaction (SIGUSR1, &strSigAction, NULL);
on rising edges.

---------------

Inside the kernel driver the IRQ will be announced with:
  s32Status = request_irq((IH_GPIO_BASE + s32AppRecvGpioNum),
                                     GpioIrqHandler,
                                     0,
                                     "GPIOINTR",
                                     NULL);

The handler looks like

static irqreturn_t GpioIrqHandler (int irq, void *dev_id)
{
    int s32GpioNumber = 0 ;
    s32GpioNumber = irq - IH_GPIO_BASE;
    set_bit (s32GpioNumber , (volatile unsigned long *) &gsts32GpioIrqStatus);
    tasklet_schedule (&GpioTasklet);
    return IRQ_HANDLED;
}

DECLARE_TASKLET(GpioTasklet, GpioIntrTasklet, 0);

------------------

What have I done wrong? Do I have a chance to speed up the response
from changing inputs?

best regards
Steffen

IRQ handler under load - slow response

[Kurt Van Dijck]

On Wed, Mar 09, 2011 at 03:02:06PM +0100, Arno Steffen wrote:

> ------------------
> 
> What have I done wrong?
You put code with different scheduling needs in the same scheduling pool.
> Do I have a chance to speed up the response
> from changing inputs?

I've been playing with scheduling policies before (I'm not the expert).
My hint:
your tasklet is executed in the softirq handler. This will run at normal
priority (see '$ ps -jefHc' for example)

a)
You could 'chrt --rr PID' of your softirqd

b)
replace your tasklet with a threaded irq handler, which you can
'chrt --rr PID'. This has the effect that only your own piece of code
is raised in priority.
> 
> best regards
> Steffen
Hope this will help.
Kurt

IRQ handler under load - slow response

[Arno Steffen]

Sorry for delay with resonse.
Thanks for your help. Most of this options for ps doesn't work on my
busybox embedded system.
Also chrt is not implemented currently. But that inspired me to
another idea. I just announced the irqs in a sperate process:
Is this what you proposed in b) ?

	i = fork();
	if (i == 0) {
		printf("Prio IRQ %d\n",getpriority(PRIO_PROCESS, getpid()));
		setpriority(PRIO_PROCESS, getpid(), -10);
		printf("Prio IRQ %d\n",getpriority(PRIO_PROCESS, getpid()));
		gpio_irq_setup(GPI_IN1, GPIOCFG_FALLINGDETECT_INTR, &irq_handler);
		gpio_irq_setup(GPI_IN2, GPIOCFG_RISINGDETECT_INTR, &irq_handler );
		printf("all IRQ handlers are setup !\n");
		do {sleep(1);} while(1);
		exit(0);
	}

This helps me reducing the delay to 2ms. Not very fast, but much
better than before.
Setting the priority doesn't have any effect by the way.

Best regards
Arno


2011/3/9 Kurt Van Dijck <kurt.van.dijck@eia.be>:
> On Wed, Mar 09, 2011 at 03:02:06PM +0100, Arno Steffen wrote:
>
>> ------------------
>>
>> What have I done wrong?
> You put code with different scheduling needs in the same scheduling pool.
>> Do I have a chance to speed up the response
>> from changing inputs?
>
> I've been playing with scheduling policies before (I'm not the expert).
> My hint:
> your tasklet is executed in the softirq handler. This will run at normal
> priority (see '$ ps -jefHc' for example)
>
> a)
> You could 'chrt --rr PID' of your softirqd
>
> b)
> replace your tasklet with a threaded irq handler, which you can
> 'chrt --rr PID'. This has the effect that only your own piece of code
> is raised in priority.
>>
>> best regards
>> Steffen
> Hope this will help.
> Kurt
>

IRQ handler under load - slow response

[Kurt Van Dijck]

On Mon, Mar 14, 2011 at 02:26:35PM +0100, Arno Steffen wrote:
> 
> Sorry for delay with resonse.
No problem.
> Thanks for your help. Most of this options for ps doesn't work on my
> busybox embedded system.
Yep, we use a regular ps for that.
> Also chrt is not implemented currently. But that inspired me to
try a program with sched_setscheduler(2)
> another idea. I just announced the irqs in a sperate process:
> Is this what you proposed in b) ?
> 
> 	i = fork();
> 	if (i == 0) {
> 		printf("Prio IRQ %d\n",getpriority(PRIO_PROCESS, getpid()));
> 		setpriority(PRIO_PROCESS, getpid(), -10);
> 		printf("Prio IRQ %d\n",getpriority(PRIO_PROCESS, getpid()));
> 		gpio_irq_setup(GPI_IN1, GPIOCFG_FALLINGDETECT_INTR, &irq_handler);
> 		gpio_irq_setup(GPI_IN2, GPIOCFG_RISINGDETECT_INTR, &irq_handler );
> 		printf("all IRQ handlers are setup !\n");
> 		do {sleep(1);} while(1);
> 		exit(0);
> 	}
Euhm, my proposed b) deals in kernel. I think your example is in userspace?
Maybe I don't understand your precise setup well.

Anyway, what I meant was something in kernel like:
  s32Status = request_irq((IH_GPIO_BASE + s32AppRecvGpioNum),
                                     GpioIrqHandler,
-				     0,
+                                     GpioIrqThread,
                                     "GPIOINTR",
                                     NULL);

static irqreturn_t GpioIrqHandler (int irq, void *dev_id)
{
    int s32GpioNumber = 0 ;
    s32GpioNumber = irq - IH_GPIO_BASE;
    set_bit (s32GpioNumber , (volatile unsigned long *) &gsts32GpioIrqStatus);
-   tasklet_schedule (&GpioTasklet);
-   return IRQ_HANDLED;
+   return IRQ_WAKE_THREAD;
}

-DECLARE_TASKLET(GpioTasklet, GpioIntrTasklet, 0);
static irqreturn_t GpioIrqThread(int irq, void *dev_id)
{
	/* do whatever the tasklet was supposed to do */
}

no, you get a seperate kernel thread that you can play with (chrt etc.).

> 
> This helps me reducing the delay to 2ms. Not very fast, but much
> better than before.
I was not aware of that there was a userspace process involved.
setting that process to SCHED_RR too (together with the kernel thread when applicable)
is a good idea.

> Setting the priority doesn't have any effect by the way.
Yep, SCHED_RR or SCHED_FIFO will do.
> 
> Best regards
> Arno

IRQ handler under load - slow response

[Arno Steffen]

2011/3/14 Kurt Van Dijck <kurt.van.dijck@eia.be>:
> On Mon, Mar 14, 2011 at 02:26:35PM +0100, Arno Steffen wrote:
>>
>> Sorry for delay with resonse.
> No problem.
>> Thanks for your help. Most of this options for ps doesn't work on my
>> busybox embedded system.
> Yep, we use a regular ps for that.
>> Also chrt is not implemented currently. But that inspired me to
> try a program with sched_setscheduler(2)
>> another idea. I just announced the irqs in a sperate process:
>> Is this what you proposed in b) ?
>>
>> ? ? ? i = fork();
>> ? ? ? if (i == 0) {
>> ? ? ? ? ? ? ? printf("Prio IRQ %d\n",getpriority(PRIO_PROCESS, getpid()));
>> ? ? ? ? ? ? ? setpriority(PRIO_PROCESS, getpid(), -10);
>> ? ? ? ? ? ? ? printf("Prio IRQ %d\n",getpriority(PRIO_PROCESS, getpid()));
>> ? ? ? ? ? ? ? gpio_irq_setup(GPI_IN1, GPIOCFG_FALLINGDETECT_INTR, &irq_handler);
>> ? ? ? ? ? ? ? gpio_irq_setup(GPI_IN2, GPIOCFG_RISINGDETECT_INTR, &irq_handler );
>> ? ? ? ? ? ? ? printf("all IRQ handlers are setup !\n");
>> ? ? ? ? ? ? ? do {sleep(1);} while(1);
>> ? ? ? ? ? ? ? exit(0);
>> ? ? ? }
> Euhm, my proposed b) deals in kernel. I think your example is in userspace?
> Maybe I don't understand your precise setup well.
>
> Anyway, what I meant was something in kernel like:
> ?s32Status = request_irq((IH_GPIO_BASE + s32AppRecvGpioNum),
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? GpioIrqHandler,
> - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?0,
> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? GpioIrqThread,
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? "GPIOINTR",
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? NULL);
>
> static irqreturn_t GpioIrqHandler (int irq, void *dev_id)
> {
> ? ?int s32GpioNumber = 0 ;
> ? ?s32GpioNumber = irq - IH_GPIO_BASE;
> ? ?set_bit (s32GpioNumber , (volatile unsigned long *) &gsts32GpioIrqStatus);
> - ? tasklet_schedule (&GpioTasklet);
> - ? return IRQ_HANDLED;
> + ? return IRQ_WAKE_THREAD;
> }
>
> -DECLARE_TASKLET(GpioTasklet, GpioIntrTasklet, 0);
> static irqreturn_t GpioIrqThread(int irq, void *dev_id)
> {
> ? ? ? ?/* do whatever the tasklet was supposed to do */
> }
>

Dear Kurt,
I did as you proposed, but doesn't have the big picture

I think there is a typo in your request_irq change:
s32Status = request_irq((IH_GPIO_BASE + s32AppRecvGpioNum),
 -                                     GpioIrqHandler,
+                                     GpioIrqThread,
                                    0,
                                     "GPIOINTR",
                                     NULL);
Is this what you mean? What you have suggested is replacing a flag by
a pointer - this cannot work. I can't see how  GpioIrqHandler and
GpioIrqThread are connected now.

> no, you get a seperate kernel thread that you can play with (chrt etc.).

I am not sure this help. The kernel thread might become a higher
proirity, but what has to be improved is that the IRQ-routine in user
space is responding faster.

>
>>
>> This helps me reducing the delay to 2ms. Not very fast, but much
>> better than before.
> I was not aware of that there was a userspace process involved.
> setting that process to SCHED_RR too (together with the kernel thread when applicable)
> is a good idea.
>
>> Setting the priority doesn't have any effect by the way.
> Yep, SCHED_RR or SCHED_FIFO will do.
>>
>> Best regards
>> Arno
>

Does it make sense to send you the hole code? I am afraid that there
is to much potential for misunderstanding. But I won't bother your.
Best regards
Arno

IRQ handler under load - slow response

[Kurt Van Dijck]

On Thu, Mar 24, 2011 at 12:56:48PM +0100, Arno Steffen wrote:
> 2011/3/14 Kurt Van Dijck <kurt.van.dijck@eia.be>:
> > On Mon, Mar 14, 2011 at 02:26:35PM +0100, Arno Steffen wrote:
> > Euhm, my proposed b) deals in kernel. I think your example is in userspace?
> > Maybe I don't understand your precise setup well.
> >
> > static irqreturn_t GpioIrqHandler (int irq, void *dev_id)
> > {
> > ? ?int s32GpioNumber = 0 ;
> > ? ?s32GpioNumber = irq - IH_GPIO_BASE;
> > ? ?set_bit (s32GpioNumber , (volatile unsigned long *) &gsts32GpioIrqStatus);
> > - ? tasklet_schedule (&GpioTasklet);
> > - ? return IRQ_HANDLED;
> > + ? return IRQ_WAKE_THREAD;
> > }
> >
> > -DECLARE_TASKLET(GpioTasklet, GpioIntrTasklet, 0);
> > static irqreturn_t GpioIrqThread(int irq, void *dev_id)
> > {
> > ? ? ? ?/* do whatever the tasklet was supposed to do */
> > }
> >
> 
> Dear Kurt,
> I did as you proposed, but doesn't have the big picture
> 
> I think there is a typo in your request_irq change:

> Is this what you mean? What you have suggested is replacing a flag by
> a pointer - this cannot work. I can't see how  GpioIrqHandler and
> GpioIrqThread are connected now.
my mistake, I had in mind to replace request_irq with 'requist_threaded_irq'.
sorry for that noise.  (I hope you did not loose too much time with that :-) ).

	s32Status = request_threaded_irq((IH_GPIO_BASE + s32AppRecvGpioNum),
			GpioIrqHandler, GpioIrqThread, 0, "GPIOINTR", NULL);

And then, you can pull your tasklet out of the softirq thread into a seperate
kernel thread (which you can set SCHED_FIFO with sched_setscheduler(2)).
> 
> I am not sure this help. The kernel thread might become a higher
> proirity, but what has to be improved is that the IRQ-routine in user
> space is responding faster.
This makes your bottom-halve respond regardless of the other load, due to the
scheduling policy.
This will not yet impact any userspace stuff. Therefore, set the scheduling
policy of you userspace task too.

In fact, the latter (setting scheduling policy of the userspace) is probably
the first step to take. put:

#include <sched.h>


int main(...) {
	const struct sched_param sched_param = {
		.sched_priority = 50,
	};
	int ret;

	ret = sched_setscheduler(0, SCHED_FIFO, &sched_param);
	if (ret < 0)
		perror(...);
	
	....
}

into your userspace program.
> 
> Does it make sense to send you the hole code? I am afraid that there
> is to much potential for misunderstanding.
This time, it was my wrong info.
I don't feel like digging in code yet, since it comprises very different parts
(kernel & userspace).
> Best regards
> Arno

IRQ handler under load - slow response

[Arno Steffen]

2011/3/24 Kurt Van Dijck <kurt.van.dijck@eia.be>:
> On Thu, Mar 24, 2011 at 12:56:48PM +0100, Arno Steffen wrote:
>> 2011/3/14 Kurt Van Dijck <kurt.van.dijck@eia.be>:
>> > On Mon, Mar 14, 2011 at 02:26:35PM +0100, Arno Steffen wrote:
>> > Euhm, my proposed b) deals in kernel. I think your example is in userspace?
>> > Maybe I don't understand your precise setup well.
>> >
>> > static irqreturn_t GpioIrqHandler (int irq, void *dev_id)
>> > {
>> > ? ?int s32GpioNumber = 0 ;
>> > ? ?s32GpioNumber = irq - IH_GPIO_BASE;
>> > ? ?set_bit (s32GpioNumber , (volatile unsigned long *) &gsts32GpioIrqStatus);
>> > - ? tasklet_schedule (&GpioTasklet);
>> > - ? return IRQ_HANDLED;
>> > + ? return IRQ_WAKE_THREAD;
>> > }
>> >
>> > -DECLARE_TASKLET(GpioTasklet, GpioIntrTasklet, 0);
>> > static irqreturn_t GpioIrqThread(int irq, void *dev_id)
>> > {
>> > ? ? ? ?/* do whatever the tasklet was supposed to do */
>> > }
>> >
>>
>> Dear Kurt,
>> I did as you proposed, but doesn't have the big picture
>>
>> I think there is a typo in your request_irq change:
>
>> Is this what you mean? What you have suggested is replacing a flag by
>> a pointer - this cannot work. I can't see how ?GpioIrqHandler and
>> GpioIrqThread are connected now.
> my mistake, I had in mind to replace request_irq with 'requist_threaded_irq'.
> sorry for that noise. ?(I hope you did not loose too much time with that :-) ).
>
> ? ? ? ?s32Status = request_threaded_irq((IH_GPIO_BASE + s32AppRecvGpioNum),
> ? ? ? ? ? ? ? ? ? ? ? ?GpioIrqHandler, GpioIrqThread, 0, "GPIOINTR", NULL);
>
> And then, you can pull your tasklet out of the softirq thread into a seperate
> kernel thread (which you can set SCHED_FIFO with sched_setscheduler(2)).
>>
>> I am not sure this help. The kernel thread might become a higher
>> proirity, but what has to be improved is that the IRQ-routine in user
>> space is responding faster.
> This makes your bottom-halve respond regardless of the other load, due to the
> scheduling policy.
> This will not yet impact any userspace stuff. Therefore, set the scheduling
> policy of you userspace task too.
>
> In fact, the latter (setting scheduling policy of the userspace) is probably
> the first step to take. put:
>
> #include <sched.h>
>
>
> int main(...) {
> ? ? ? ?const struct sched_param sched_param = {
> ? ? ? ? ? ? ? ?.sched_priority = 50,
> ? ? ? ?};
> ? ? ? ?int ret;
>
> ? ? ? ?ret = sched_setscheduler(0, SCHED_FIFO, &sched_param);
> ? ? ? ?if (ret < 0)
> ? ? ? ? ? ? ? ?perror(...);
>
> ? ? ? ?....
> }
>
> into your userspace program.
>>
>> Does it make sense to send you the hole code? I am afraid that there
>> is to much potential for misunderstanding.
> This time, it was my wrong info.
> I don't feel like digging in code yet, since it comprises very different parts
> (kernel & userspace).
>> Best regards
>> Arno

I get it working with your help. I see now threads for every IRQ I
anounce to the system. Nevertheless it doesn't change the behaviour,
even after increasing their prio. My assumption is: The kernel IRQs
threads are processed fast, I do have the trouble on the user side of
the IRQ handler.

The best performance I get, if all processes on my device are running
in policy OTHER. As soon as my code is running in FIFO or RR I have
problems. If the IRQs-thread has higher priority it works better, but
is far away from reliability of OTHER policy. This is in opposite to
all what I know about scheduling policies.

Best regards
Arno

IRQ handler under load - slow response

[Kurt Van Dijck]

On Fri, Mar 25, 2011 at 09:46:43AM +0100, Arno Steffen wrote:
> 2011/3/24 Kurt Van Dijck <kurt.van.dijck@eia.be>:
> > On Thu, Mar 24, 2011 at 12:56:48PM +0100, Arno Steffen wrote:
> >> 2011/3/14 Kurt Van Dijck <kurt.van.dijck@eia.be>:
> >> > On Mon, Mar 14, 2011 at 02:26:35PM +0100, Arno Steffen wrote:
> 
> I get it working with your help. I see now threads for every IRQ I
> anounce to the system.
Good news. I'm in fact a bit curious to the results...
> Nevertheless it doesn't change the behaviour,
> even after increasing their prio. My assumption is: The kernel IRQs
> threads are processed fast, I do have the trouble on the user side of
> the IRQ handler.
> 
> The best performance I get, if all processes on my device are running
> in policy OTHER.
Just to update my memory: I think threaded-irq will give:
* less peak perfomance
* less average performance
* better 'worst' performance
ie. There should be much less jitter in the performance measurement.
> As soon as my code is running in FIFO or RR I have
> problems. If the IRQs-thread has higher priority it works better, but
> is far away from reliability of OTHER policy.
That's less good news.
I suppose you have enabled all relevant RT & preemptive options.
Then I'm out of ideas too.
> This is in opposite to
> all what I know about scheduling policies.
Ack.
> 
> Best regards
> Arno

Regards,
Kurt