[RFC0/1] megaraid_sas : IRQ polling using threaded interrupt

[RFC0/1] megaraid_sas : IRQ polling using threaded interrupt

[Kashyap Desai]

Hi,

I refer below thread for interrupt polling related discussion for
storage devices (NVME/HBA).

 http://lists.infradead.org/pipermail/linux-nvme/2017-January/007749.html

 I am trying to evaluate the similar concept to reduce interrupts
using irq-poll and threaded ISR. Below is high level changes I did in
my experiment.


In megaraid_sas driver, I schedule irq poll using =E2=80=9Cirq_poll_sched=
=E2=80=9D and
disable that particular IRQ line. I was expecting more completion
after I disable irq line, but in most of the cases  just one
completion happened from irq poll context. It must be due to host side
processing is much faster than IO processing at back end device.
Similar observation was posted in above mentioned thread as well. I
added manual wait using =E2=80=9Cudelay()=E2=80=9D and wait for some more t=
ime in irq
poll context. Using additional wait, I am able to reduce interrupt per
seconds and performance impact on latency is not visible since I
choose udelay(1). One drawback I see is overall CPU utilization goes
up since driver is using extra delay to reduce interrupts.

 To overcome CPU utilization issue, I switch to threaded ISR and
replace udelay() with usleep().  Using threaded interrupt based
polling CPU utilization goes to normal.

 I am trying to understand what is a drawback using threaded ISR and
doing interrupt polling from threaded ISR.  One problem I understood
is threaded ISR can be preempted by other high priority context (most
likely soft/hard interrupt).  If threaded interrupt is running on some
CPU-x and some other device is also requesting interrupt on same
CPU-x, there can be latency introduced because CPU-x will complete ISR
before it can run threaded ISR.

Thanks ,Kashyap

Re: [RFC0/1] megaraid_sas : IRQ polling using threaded interrupt

[Sagi Grimberg]

> Hi,
> 
> I refer below thread for interrupt polling related discussion for
> storage devices (NVME/HBA).
> 
>   http://lists.infradead.org/pipermail/linux-nvme/2017-January/007749.html
> 
>   I am trying to evaluate the similar concept to reduce interrupts
> using irq-poll and threaded ISR. Below is high level changes I did in
> my experiment.

You want to reduce interrupts because you want to fix a bug where
your driver never exit from hard-irq? or trying to optimize the
efficiency of the system?

> In megaraid_sas driver, I schedule irq poll using “irq_poll_sched” and
> disable that particular IRQ line. I was expecting more completion
> after I disable irq line, but in most of the cases  just one
> completion happened from irq poll context. It must be due to host side
> processing is much faster than IO processing at back end device.
> Similar observation was posted in above mentioned thread as well.

Is your device able to coalesce interrupts?

> I added manual wait using “udelay()” and wait for some more time in irq
> poll context. Using additional wait, I am able to reduce interrupt per
> seconds and performance impact on latency is not visible since I
> choose udelay(1). One drawback I see is overall CPU utilization goes
> up since driver is using extra delay to reduce interrupts.
> 
>   To overcome CPU utilization issue, I switch to threaded ISR and
> replace udelay() with usleep().  Using threaded interrupt based
> polling CPU utilization goes to normal.

Doesn't sound like the right approach to me, but I don't understand
what you are trying to achieve.

RE: [RFC0/1] megaraid_sas : IRQ polling using threaded interrupt

[Kashyap Desai]

> >
> >   I am trying to evaluate the similar concept to reduce interrupts
> > using irq-poll and threaded ISR. Below is high level changes I did in
> > my experiment.
>
> You want to reduce interrupts because you want to fix a bug where
> your driver never exit from hard-irq? or trying to optimize the
> efficiency of the system?

We have already adopted irq-poll interface to avoid any hard-lockup issue.
This time, I want to evaluate multiple options to reduce interrupts per
seconds for MegaRaid and HBA.
In certain cases, we observe that IOPs are not scaling mainly because of
each IO completion happens with one interrupt.

>
> > In megaraid_sas driver, I schedule irq poll using =E2=80=9Cirq_poll_sch=
ed=E2=80=9D and
> > disable that particular IRQ line. I was expecting more completion
> > after I disable irq line, but in most of the cases  just one
> > completion happened from irq poll context. It must be due to host side
> > processing is much faster than IO processing at back end device.
> > Similar observation was posted in above mentioned thread as well.
>
> Is your device able to coalesce interrupts?

Yes, our device is capable of doing interrupt coalescing. That interface is
already in plan for this exercise.  Disadvantage of allowing h/w doing
interrupt coalescing is low QD profiles. We see latency goes high upto 20%
and we want to avoid that as well.

>
> > I added manual wait using =E2=80=9Cudelay()=E2=80=9D and wait for some =
more time in irq
> > poll context. Using additional wait, I am able to reduce interrupt per
> > seconds and performance impact on latency is not visible since I
> > choose udelay(1). One drawback I see is overall CPU utilization goes
> > up since driver is using extra delay to reduce interrupts.
> >
> >   To overcome CPU utilization issue, I switch to threaded ISR and
> > replace udelay() with usleep().  Using threaded interrupt based
> > polling CPU utilization goes to normal.
>
> Doesn't sound like the right approach to me, but I don't understand
> what you are trying to achieve.

I posted patch with threaded ISR to do interrupt polling instead of taking
advantage of irq-poll. I can do same thing with irq-poll as well.  In eithe=
r
approach, I found that just doing interrupt disable and polling reply queue
processing does not server purpose. Every time I disable interrupt
(disable_irq_nosync) and start polling of the reply queue, it quits
immediately since CPU processing is much faster compare to IO completion
from backend. Eventually, I have to spend some time after
"disable_irq_nosync" to gain benefit of the polling.  I can not use
sleepable delay in irq-poll, but I can use sleepable delay in threaded isr.
This difference comes with pros and cons. Irq-poll based interface hogs mor=
e
CPU whereas threded isr avoid cpu hogging. I still think that thread-isr is
not a good idea because of fairness as you explained earlier in mentioned
discussion.

My confusion and question is - "threaded ISR is viable option to use for
interrupt polling or irq-poll is suitable design? "