* [PATCH 2/2] staging: octeon-ethernet: allow to use only 1 CPU for packet processing
2013-09-28 19:50 [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special Aaro Koskinen
@ 2013-09-28 19:50 ` Aaro Koskinen
2013-09-28 20:40 ` [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special Richard Weinberger
` (3 subsequent siblings)
4 siblings, 0 replies; 15+ messages in thread
From: Aaro Koskinen @ 2013-09-28 19:50 UTC (permalink / raw)
To: devel, linux-mips, Greg Kroah-Hartman, David Daney; +Cc: richard, Aaro Koskinen
Module parameter max_rx_cpus has off-by-one error. Fix that.
Signed-off-by: Aaro Koskinen <aaro.koskinen@iki.fi>
---
drivers/staging/octeon/ethernet-rx.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/staging/octeon/ethernet-rx.c b/drivers/staging/octeon/ethernet-rx.c
index de831c1..fe1ee49 100644
--- a/drivers/staging/octeon/ethernet-rx.c
+++ b/drivers/staging/octeon/ethernet-rx.c
@@ -513,7 +513,7 @@ void cvm_oct_rx_initialize(void)
if (NULL == dev_for_napi)
panic("No net_devices were allocated.");
- if (max_rx_cpus > 1 && max_rx_cpus < num_online_cpus())
+ if (max_rx_cpus >= 1 && max_rx_cpus < num_online_cpus())
atomic_set(&core_state.available_cores, max_rx_cpus);
else
atomic_set(&core_state.available_cores, num_online_cpus());
--
1.8.4.rc3
^ permalink raw reply related [flat|nested] 15+ messages in thread* Re: [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special
2013-09-28 19:50 [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special Aaro Koskinen
2013-09-28 19:50 ` [PATCH 2/2] staging: octeon-ethernet: allow to use only 1 CPU for packet processing Aaro Koskinen
@ 2013-09-28 20:40 ` Richard Weinberger
2013-09-28 21:12 ` Aaro Koskinen
2013-09-28 21:15 ` Richard Weinberger
` (2 subsequent siblings)
4 siblings, 1 reply; 15+ messages in thread
From: Richard Weinberger @ 2013-09-28 20:40 UTC (permalink / raw)
To: Aaro Koskinen; +Cc: devel, linux-mips, Greg Kroah-Hartman, David Daney
Am 28.09.2013 21:50, schrieb Aaro Koskinen:
> Currently the driver assumes that CPU 0 is handling all the hard IRQs.
> This is wrong in Linux SMP systems where user is allowed to assign to
> hardware IRQs to any CPU. The driver will stop working if user sets
> smp_affinity so that interrupts end up being handled by other than CPU
> 0. The patch fixes that.
>
> Signed-off-by: Aaro Koskinen <aaro.koskinen@iki.fi>
> ---
> drivers/staging/octeon/ethernet-rx.c | 12 ++++++------
> 1 file changed, 6 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/staging/octeon/ethernet-rx.c b/drivers/staging/octeon/ethernet-rx.c
> index e14a1bb..de831c1 100644
> --- a/drivers/staging/octeon/ethernet-rx.c
> +++ b/drivers/staging/octeon/ethernet-rx.c
> @@ -80,6 +80,8 @@ struct cvm_oct_core_state {
>
> static struct cvm_oct_core_state core_state __cacheline_aligned_in_smp;
>
> +static int cvm_irq_cpu = -1;
Why are you introducing a new global variable here?
Can't you pass cvm_irq_cpu as argument to cvm_oct_enable_napi()?
Thanks,
//richard
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special
2013-09-28 20:40 ` [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special Richard Weinberger
@ 2013-09-28 21:12 ` Aaro Koskinen
0 siblings, 0 replies; 15+ messages in thread
From: Aaro Koskinen @ 2013-09-28 21:12 UTC (permalink / raw)
To: Richard Weinberger; +Cc: devel, linux-mips, Greg Kroah-Hartman, David Daney
Hi,
On Sat, Sep 28, 2013 at 10:40:47PM +0200, Richard Weinberger wrote:
> Am 28.09.2013 21:50, schrieb Aaro Koskinen:
> > Currently the driver assumes that CPU 0 is handling all the hard IRQs.
> > This is wrong in Linux SMP systems where user is allowed to assign to
> > hardware IRQs to any CPU. The driver will stop working if user sets
> > smp_affinity so that interrupts end up being handled by other than CPU
> > 0. The patch fixes that.
> >
> > Signed-off-by: Aaro Koskinen <aaro.koskinen@iki.fi>
> > ---
> > drivers/staging/octeon/ethernet-rx.c | 12 ++++++------
> > 1 file changed, 6 insertions(+), 6 deletions(-)
> >
> > diff --git a/drivers/staging/octeon/ethernet-rx.c b/drivers/staging/octeon/ethernet-rx.c
> > index e14a1bb..de831c1 100644
> > --- a/drivers/staging/octeon/ethernet-rx.c
> > +++ b/drivers/staging/octeon/ethernet-rx.c
> > @@ -80,6 +80,8 @@ struct cvm_oct_core_state {
> >
> > static struct cvm_oct_core_state core_state __cacheline_aligned_in_smp;
> >
> > +static int cvm_irq_cpu = -1;
>
> Why are you introducing a new global variable here?
> Can't you pass cvm_irq_cpu as argument to cvm_oct_enable_napi()?
This information needs to be accessed in cvm_oct_no_more_work().
Maybe I'm missing something obvious, but I don't get how the argument
could be accessed in or passed to napi poll routine which is calling
cvm_oct_no_more_work()?
A.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special
2013-09-28 19:50 [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special Aaro Koskinen
2013-09-28 19:50 ` [PATCH 2/2] staging: octeon-ethernet: allow to use only 1 CPU for packet processing Aaro Koskinen
2013-09-28 20:40 ` [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special Richard Weinberger
@ 2013-09-28 21:15 ` Richard Weinberger
2013-09-30 17:23 ` David Daney
2013-10-03 20:36 ` Greg Kroah-Hartman
4 siblings, 0 replies; 15+ messages in thread
From: Richard Weinberger @ 2013-09-28 21:15 UTC (permalink / raw)
To: Aaro Koskinen; +Cc: devel, linux-mips, Greg Kroah-Hartman, David Daney
Am 28.09.2013 21:50, schrieb Aaro Koskinen:
> Currently the driver assumes that CPU 0 is handling all the hard IRQs.
> This is wrong in Linux SMP systems where user is allowed to assign to
> hardware IRQs to any CPU. The driver will stop working if user sets
> smp_affinity so that interrupts end up being handled by other than CPU
> 0. The patch fixes that.
You are right, sorry. I somehow mixed up the function names.
Thanks,
//richard
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special
@ 2013-09-30 17:23 ` David Daney
0 siblings, 0 replies; 15+ messages in thread
From: David Daney @ 2013-09-30 17:23 UTC (permalink / raw)
To: Aaro Koskinen; +Cc: devel, linux-mips, Greg Kroah-Hartman, David Daney, richard
On 09/28/2013 12:50 PM, Aaro Koskinen wrote:
> Currently the driver assumes that CPU 0 is handling all the hard IRQs.
> This is wrong in Linux SMP systems where user is allowed to assign to
> hardware IRQs to any CPU. The driver will stop working if user sets
> smp_affinity so that interrupts end up being handled by other than CPU
> 0. The patch fixes that.
>
> Signed-off-by: Aaro Koskinen <aaro.koskinen@iki.fi>
> ---
> drivers/staging/octeon/ethernet-rx.c | 12 ++++++------
> 1 file changed, 6 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/staging/octeon/ethernet-rx.c b/drivers/staging/octeon/ethernet-rx.c
> index e14a1bb..de831c1 100644
> --- a/drivers/staging/octeon/ethernet-rx.c
> +++ b/drivers/staging/octeon/ethernet-rx.c
> @@ -80,6 +80,8 @@ struct cvm_oct_core_state {
>
> static struct cvm_oct_core_state core_state __cacheline_aligned_in_smp;
>
> +static int cvm_irq_cpu = -1;
> +
> static void cvm_oct_enable_napi(void *_)
> {
> int cpu = smp_processor_id();
> @@ -112,11 +114,7 @@ static void cvm_oct_no_more_work(void)
> {
> int cpu = smp_processor_id();
>
> - /*
> - * CPU zero is special. It always has the irq enabled when
> - * waiting for incoming packets.
> - */
> - if (cpu == 0) {
> + if (cpu == cvm_irq_cpu) {
> enable_irq(OCTEON_IRQ_WORKQ0 + pow_receive_group);
> return;
> }
> @@ -135,6 +133,7 @@ static irqreturn_t cvm_oct_do_interrupt(int cpl, void *dev_id)
> {
> /* Disable the IRQ and start napi_poll. */
> disable_irq_nosync(OCTEON_IRQ_WORKQ0 + pow_receive_group);
> + cvm_irq_cpu = smp_processor_id();
> cvm_oct_enable_napi(NULL);
>
> return IRQ_HANDLED;
> @@ -547,8 +546,9 @@ void cvm_oct_rx_initialize(void)
> cvmx_write_csr(CVMX_POW_WQ_INT_PC, int_pc.u64);
>
>
> - /* Scheduld NAPI now. This will indirectly enable interrupts. */
> + /* Schedule NAPI now. */
> cvm_oct_enable_one_cpu();
> + enable_irq(OCTEON_IRQ_WORKQ0 + pow_receive_group);
The fact that you have to manually enable irqs here indicates that the
patch is not good.
Either the enable_irq() is unnecessary, or you broke the logic for
enabling NAPI on more than one CPU.
I am not sure which is the case, but I think it would be best if you
supplied a fixed patch set that corrects whichever happens to be the case.
David Daney
> }
>
> void cvm_oct_rx_shutdown(void)
>
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special
@ 2013-09-30 17:23 ` David Daney
0 siblings, 0 replies; 15+ messages in thread
From: David Daney @ 2013-09-30 17:23 UTC (permalink / raw)
To: Aaro Koskinen; +Cc: devel, linux-mips, Greg Kroah-Hartman, David Daney, richard
On 09/28/2013 12:50 PM, Aaro Koskinen wrote:
> Currently the driver assumes that CPU 0 is handling all the hard IRQs.
> This is wrong in Linux SMP systems where user is allowed to assign to
> hardware IRQs to any CPU. The driver will stop working if user sets
> smp_affinity so that interrupts end up being handled by other than CPU
> 0. The patch fixes that.
>
> Signed-off-by: Aaro Koskinen <aaro.koskinen@iki.fi>
> ---
> drivers/staging/octeon/ethernet-rx.c | 12 ++++++------
> 1 file changed, 6 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/staging/octeon/ethernet-rx.c b/drivers/staging/octeon/ethernet-rx.c
> index e14a1bb..de831c1 100644
> --- a/drivers/staging/octeon/ethernet-rx.c
> +++ b/drivers/staging/octeon/ethernet-rx.c
> @@ -80,6 +80,8 @@ struct cvm_oct_core_state {
>
> static struct cvm_oct_core_state core_state __cacheline_aligned_in_smp;
>
> +static int cvm_irq_cpu = -1;
> +
> static void cvm_oct_enable_napi(void *_)
> {
> int cpu = smp_processor_id();
> @@ -112,11 +114,7 @@ static void cvm_oct_no_more_work(void)
> {
> int cpu = smp_processor_id();
>
> - /*
> - * CPU zero is special. It always has the irq enabled when
> - * waiting for incoming packets.
> - */
> - if (cpu == 0) {
> + if (cpu == cvm_irq_cpu) {
> enable_irq(OCTEON_IRQ_WORKQ0 + pow_receive_group);
> return;
> }
> @@ -135,6 +133,7 @@ static irqreturn_t cvm_oct_do_interrupt(int cpl, void *dev_id)
> {
> /* Disable the IRQ and start napi_poll. */
> disable_irq_nosync(OCTEON_IRQ_WORKQ0 + pow_receive_group);
> + cvm_irq_cpu = smp_processor_id();
> cvm_oct_enable_napi(NULL);
>
> return IRQ_HANDLED;
> @@ -547,8 +546,9 @@ void cvm_oct_rx_initialize(void)
> cvmx_write_csr(CVMX_POW_WQ_INT_PC, int_pc.u64);
>
>
> - /* Scheduld NAPI now. This will indirectly enable interrupts. */
> + /* Schedule NAPI now. */
> cvm_oct_enable_one_cpu();
> + enable_irq(OCTEON_IRQ_WORKQ0 + pow_receive_group);
The fact that you have to manually enable irqs here indicates that the
patch is not good.
Either the enable_irq() is unnecessary, or you broke the logic for
enabling NAPI on more than one CPU.
I am not sure which is the case, but I think it would be best if you
supplied a fixed patch set that corrects whichever happens to be the case.
David Daney
> }
>
> void cvm_oct_rx_shutdown(void)
>
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special
2013-09-30 17:23 ` David Daney
(?)
@ 2013-09-30 19:35 ` Aaro Koskinen
2013-09-30 19:41 ` David Daney
-1 siblings, 1 reply; 15+ messages in thread
From: Aaro Koskinen @ 2013-09-30 19:35 UTC (permalink / raw)
To: David Daney; +Cc: devel, linux-mips, Greg Kroah-Hartman, David Daney, richard
Hi,
On Mon, Sep 30, 2013 at 10:23:10AM -0700, David Daney wrote:
> On 09/28/2013 12:50 PM, Aaro Koskinen wrote:
> >Currently the driver assumes that CPU 0 is handling all the hard IRQs.
> >This is wrong in Linux SMP systems where user is allowed to assign to
> >hardware IRQs to any CPU. The driver will stop working if user sets
> >smp_affinity so that interrupts end up being handled by other than CPU
> >0. The patch fixes that.
> >
> >Signed-off-by: Aaro Koskinen <aaro.koskinen@iki.fi>
> >---
> > drivers/staging/octeon/ethernet-rx.c | 12 ++++++------
> > 1 file changed, 6 insertions(+), 6 deletions(-)
> >
> >diff --git a/drivers/staging/octeon/ethernet-rx.c b/drivers/staging/octeon/ethernet-rx.c
> >index e14a1bb..de831c1 100644
> >--- a/drivers/staging/octeon/ethernet-rx.c
> >+++ b/drivers/staging/octeon/ethernet-rx.c
> >@@ -80,6 +80,8 @@ struct cvm_oct_core_state {
> >
> > static struct cvm_oct_core_state core_state __cacheline_aligned_in_smp;
> >
> >+static int cvm_irq_cpu = -1;
> >+
> > static void cvm_oct_enable_napi(void *_)
> > {
> > int cpu = smp_processor_id();
> >@@ -112,11 +114,7 @@ static void cvm_oct_no_more_work(void)
> > {
> > int cpu = smp_processor_id();
> >
> >- /*
> >- * CPU zero is special. It always has the irq enabled when
> >- * waiting for incoming packets.
> >- */
> >- if (cpu == 0) {
> >+ if (cpu == cvm_irq_cpu) {
> > enable_irq(OCTEON_IRQ_WORKQ0 + pow_receive_group);
> > return;
> > }
> >@@ -135,6 +133,7 @@ static irqreturn_t cvm_oct_do_interrupt(int cpl, void *dev_id)
> > {
> > /* Disable the IRQ and start napi_poll. */
> > disable_irq_nosync(OCTEON_IRQ_WORKQ0 + pow_receive_group);
> >+ cvm_irq_cpu = smp_processor_id();
> > cvm_oct_enable_napi(NULL);
> >
> > return IRQ_HANDLED;
> >@@ -547,8 +546,9 @@ void cvm_oct_rx_initialize(void)
> > cvmx_write_csr(CVMX_POW_WQ_INT_PC, int_pc.u64);
> >
> >
> >- /* Scheduld NAPI now. This will indirectly enable interrupts. */
> >+ /* Schedule NAPI now. */
> > cvm_oct_enable_one_cpu();
> >+ enable_irq(OCTEON_IRQ_WORKQ0 + pow_receive_group);
>
> The fact that you have to manually enable irqs here indicates that
> the patch is not good.
>
> Either the enable_irq() is unnecessary, or you broke the logic for
> enabling NAPI on more than one CPU.
>
> I am not sure which is the case, but I think it would be best if you
> supplied a fixed patch set that corrects whichever happens to be the
> case.
No, the original logic was already broken. The code assumed that the
NAPI scheduled by the driver init gets executed always on CPU 0. The
IRQ got enabled just because we are lucky.
The patch removes such assumption. During the driver init, the IRQ is
disabled and cvm_irq_cpu is -1. So when the NAPI scheduled (on whatever
CPU) by the init is done, the check in cvm_oct_no_more_work() will be
false and the IRQ remains disabled. So the init routine has to enable IRQ
"manually". This is a special case only for the driver initialization.
During the normal operation, the IRQ handler record the CPU (on which
it's going the schedule for the first NAPI poll) in cvm_irq_cpu so
cvm_oct_no_more_work() will always re-enable the IRQ correctly.
A.
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special
@ 2013-09-30 19:41 ` David Daney
0 siblings, 0 replies; 15+ messages in thread
From: David Daney @ 2013-09-30 19:41 UTC (permalink / raw)
To: Aaro Koskinen; +Cc: devel, linux-mips, Greg Kroah-Hartman, David Daney, richard
On 09/30/2013 12:35 PM, Aaro Koskinen wrote:
> Hi,
>
> On Mon, Sep 30, 2013 at 10:23:10AM -0700, David Daney wrote:
>> On 09/28/2013 12:50 PM, Aaro Koskinen wrote:
>>> Currently the driver assumes that CPU 0 is handling all the hard IRQs.
>>> This is wrong in Linux SMP systems where user is allowed to assign to
>>> hardware IRQs to any CPU. The driver will stop working if user sets
>>> smp_affinity so that interrupts end up being handled by other than CPU
>>> 0. The patch fixes that.
>>>
>>> Signed-off-by: Aaro Koskinen <aaro.koskinen@iki.fi>
>>> ---
>>> drivers/staging/octeon/ethernet-rx.c | 12 ++++++------
>>> 1 file changed, 6 insertions(+), 6 deletions(-)
>>>
>>> diff --git a/drivers/staging/octeon/ethernet-rx.c b/drivers/staging/octeon/ethernet-rx.c
>>> index e14a1bb..de831c1 100644
>>> --- a/drivers/staging/octeon/ethernet-rx.c
>>> +++ b/drivers/staging/octeon/ethernet-rx.c
>>> @@ -80,6 +80,8 @@ struct cvm_oct_core_state {
>>>
>>> static struct cvm_oct_core_state core_state __cacheline_aligned_in_smp;
>>>
>>> +static int cvm_irq_cpu = -1;
>>> +
>>> static void cvm_oct_enable_napi(void *_)
>>> {
>>> int cpu = smp_processor_id();
>>> @@ -112,11 +114,7 @@ static void cvm_oct_no_more_work(void)
>>> {
>>> int cpu = smp_processor_id();
>>>
>>> - /*
>>> - * CPU zero is special. It always has the irq enabled when
>>> - * waiting for incoming packets.
>>> - */
>>> - if (cpu == 0) {
>>> + if (cpu == cvm_irq_cpu) {
>>> enable_irq(OCTEON_IRQ_WORKQ0 + pow_receive_group);
>>> return;
>>> }
>>> @@ -135,6 +133,7 @@ static irqreturn_t cvm_oct_do_interrupt(int cpl, void *dev_id)
>>> {
>>> /* Disable the IRQ and start napi_poll. */
>>> disable_irq_nosync(OCTEON_IRQ_WORKQ0 + pow_receive_group);
>>> + cvm_irq_cpu = smp_processor_id();
>>> cvm_oct_enable_napi(NULL);
>>>
>>> return IRQ_HANDLED;
>>> @@ -547,8 +546,9 @@ void cvm_oct_rx_initialize(void)
>>> cvmx_write_csr(CVMX_POW_WQ_INT_PC, int_pc.u64);
>>>
>>>
>>> - /* Scheduld NAPI now. This will indirectly enable interrupts. */
>>> + /* Schedule NAPI now. */
>>> cvm_oct_enable_one_cpu();
>>> + enable_irq(OCTEON_IRQ_WORKQ0 + pow_receive_group);
>>
>> The fact that you have to manually enable irqs here indicates that
>> the patch is not good.
>>
>> Either the enable_irq() is unnecessary, or you broke the logic for
>> enabling NAPI on more than one CPU.
>>
>> I am not sure which is the case, but I think it would be best if you
>> supplied a fixed patch set that corrects whichever happens to be the
>> case.
>
> No, the original logic was already broken. The code assumed that the
> NAPI scheduled by the driver init gets executed always on CPU 0. The
> IRQ got enabled just because we are lucky.
No. The default affinity for all irqs is CPU0 for just this reason. So
there was no luck involved.
>
> The patch removes such assumption. During the driver init, the IRQ is
> disabled and cvm_irq_cpu is -1. So when the NAPI scheduled (on whatever
> CPU) by the init is done, the check in cvm_oct_no_more_work() will be
> false and the IRQ remains disabled. So the init routine has to enable IRQ
> "manually". This is a special case only for the driver initialization.
No. We schedule NAPI at this point. If there are no packets available,
the normal driver logic enables the irq. There is no need to explicitly
enable it in cvm_oct_rx_initialize().
If this doesn't work, and you need to have an additional place that you
enable the irq, then you broke something.
>
> During the normal operation, the IRQ handler record the CPU (on which
> it's going the schedule for the first NAPI poll) in cvm_irq_cpu so
> cvm_oct_no_more_work() will always re-enable the IRQ correctly.
>
> A.
>
>
>
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special
@ 2013-09-30 19:41 ` David Daney
0 siblings, 0 replies; 15+ messages in thread
From: David Daney @ 2013-09-30 19:41 UTC (permalink / raw)
To: Aaro Koskinen; +Cc: devel, linux-mips, Greg Kroah-Hartman, David Daney, richard
On 09/30/2013 12:35 PM, Aaro Koskinen wrote:
> Hi,
>
> On Mon, Sep 30, 2013 at 10:23:10AM -0700, David Daney wrote:
>> On 09/28/2013 12:50 PM, Aaro Koskinen wrote:
>>> Currently the driver assumes that CPU 0 is handling all the hard IRQs.
>>> This is wrong in Linux SMP systems where user is allowed to assign to
>>> hardware IRQs to any CPU. The driver will stop working if user sets
>>> smp_affinity so that interrupts end up being handled by other than CPU
>>> 0. The patch fixes that.
>>>
>>> Signed-off-by: Aaro Koskinen <aaro.koskinen@iki.fi>
>>> ---
>>> drivers/staging/octeon/ethernet-rx.c | 12 ++++++------
>>> 1 file changed, 6 insertions(+), 6 deletions(-)
>>>
>>> diff --git a/drivers/staging/octeon/ethernet-rx.c b/drivers/staging/octeon/ethernet-rx.c
>>> index e14a1bb..de831c1 100644
>>> --- a/drivers/staging/octeon/ethernet-rx.c
>>> +++ b/drivers/staging/octeon/ethernet-rx.c
>>> @@ -80,6 +80,8 @@ struct cvm_oct_core_state {
>>>
>>> static struct cvm_oct_core_state core_state __cacheline_aligned_in_smp;
>>>
>>> +static int cvm_irq_cpu = -1;
>>> +
>>> static void cvm_oct_enable_napi(void *_)
>>> {
>>> int cpu = smp_processor_id();
>>> @@ -112,11 +114,7 @@ static void cvm_oct_no_more_work(void)
>>> {
>>> int cpu = smp_processor_id();
>>>
>>> - /*
>>> - * CPU zero is special. It always has the irq enabled when
>>> - * waiting for incoming packets.
>>> - */
>>> - if (cpu == 0) {
>>> + if (cpu == cvm_irq_cpu) {
>>> enable_irq(OCTEON_IRQ_WORKQ0 + pow_receive_group);
>>> return;
>>> }
>>> @@ -135,6 +133,7 @@ static irqreturn_t cvm_oct_do_interrupt(int cpl, void *dev_id)
>>> {
>>> /* Disable the IRQ and start napi_poll. */
>>> disable_irq_nosync(OCTEON_IRQ_WORKQ0 + pow_receive_group);
>>> + cvm_irq_cpu = smp_processor_id();
>>> cvm_oct_enable_napi(NULL);
>>>
>>> return IRQ_HANDLED;
>>> @@ -547,8 +546,9 @@ void cvm_oct_rx_initialize(void)
>>> cvmx_write_csr(CVMX_POW_WQ_INT_PC, int_pc.u64);
>>>
>>>
>>> - /* Scheduld NAPI now. This will indirectly enable interrupts. */
>>> + /* Schedule NAPI now. */
>>> cvm_oct_enable_one_cpu();
>>> + enable_irq(OCTEON_IRQ_WORKQ0 + pow_receive_group);
>>
>> The fact that you have to manually enable irqs here indicates that
>> the patch is not good.
>>
>> Either the enable_irq() is unnecessary, or you broke the logic for
>> enabling NAPI on more than one CPU.
>>
>> I am not sure which is the case, but I think it would be best if you
>> supplied a fixed patch set that corrects whichever happens to be the
>> case.
>
> No, the original logic was already broken. The code assumed that the
> NAPI scheduled by the driver init gets executed always on CPU 0. The
> IRQ got enabled just because we are lucky.
No. The default affinity for all irqs is CPU0 for just this reason. So
there was no luck involved.
>
> The patch removes such assumption. During the driver init, the IRQ is
> disabled and cvm_irq_cpu is -1. So when the NAPI scheduled (on whatever
> CPU) by the init is done, the check in cvm_oct_no_more_work() will be
> false and the IRQ remains disabled. So the init routine has to enable IRQ
> "manually". This is a special case only for the driver initialization.
No. We schedule NAPI at this point. If there are no packets available,
the normal driver logic enables the irq. There is no need to explicitly
enable it in cvm_oct_rx_initialize().
If this doesn't work, and you need to have an additional place that you
enable the irq, then you broke something.
>
> During the normal operation, the IRQ handler record the CPU (on which
> it's going the schedule for the first NAPI poll) in cvm_irq_cpu so
> cvm_oct_no_more_work() will always re-enable the IRQ correctly.
>
> A.
>
>
>
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special
2013-09-30 19:41 ` David Daney
(?)
@ 2013-09-30 19:56 ` Aaro Koskinen
2013-09-30 21:08 ` David Daney
-1 siblings, 1 reply; 15+ messages in thread
From: Aaro Koskinen @ 2013-09-30 19:56 UTC (permalink / raw)
To: David Daney; +Cc: devel, linux-mips, David Daney, richard, Greg Kroah-Hartman
Hi,
On Mon, Sep 30, 2013 at 12:41:59PM -0700, David Daney wrote:
> On 09/30/2013 12:35 PM, Aaro Koskinen wrote:
> >No, the original logic was already broken. The code assumed that the
> >NAPI scheduled by the driver init gets executed always on CPU 0. The
> >IRQ got enabled just because we are lucky.
>
> No. The default affinity for all irqs is CPU0 for just this reason.
> So there was no luck involved.
According the Kconfig, this driver can be compiled as a module:
> config OCTEON_ETHERNET
> tristate "Cavium Networks Octeon Ethernet support"
[...]
> To compile this driver as a module, choose M here. The module
> will be called octeon-ethernet.
What guarantees that CPU0 is around (or the smp_affinity is at its
default value) by the time user executes modprobe?
A.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special
@ 2013-09-30 21:08 ` David Daney
0 siblings, 0 replies; 15+ messages in thread
From: David Daney @ 2013-09-30 21:08 UTC (permalink / raw)
To: Aaro Koskinen; +Cc: devel, linux-mips, David Daney, richard, Greg Kroah-Hartman
On 09/30/2013 12:56 PM, Aaro Koskinen wrote:
> Hi,
>
> On Mon, Sep 30, 2013 at 12:41:59PM -0700, David Daney wrote:
>> On 09/30/2013 12:35 PM, Aaro Koskinen wrote:
>>> No, the original logic was already broken. The code assumed that the
>>> NAPI scheduled by the driver init gets executed always on CPU 0. The
>>> IRQ got enabled just because we are lucky.
>>
>> No. The default affinity for all irqs is CPU0 for just this reason.
>> So there was no luck involved.
>
> According the Kconfig, this driver can be compiled as a module:
>
>> config OCTEON_ETHERNET
>> tristate "Cavium Networks Octeon Ethernet support"
> [...]
>> To compile this driver as a module, choose M here. The module
>> will be called octeon-ethernet.
>
> What guarantees that CPU0 is around (or the smp_affinity is at its
> default value) by the time user executes modprobe?
Nothing enforced by the kernel. Just don't take CPU0 off-line and you
should be good to go.
There is a lot of room for improvement in the driver.
Really this whole thing of starting NAPI on multiple CPUs for the same
input queue is not good. It leads to loss of packet ordering when
forwarding.
What we really want is to have one POW group (input queue) per CPU and
only run one CPU per group. If we did that, we wouldn't have all these
affinity issues.
David Daney
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special
@ 2013-09-30 21:08 ` David Daney
0 siblings, 0 replies; 15+ messages in thread
From: David Daney @ 2013-09-30 21:08 UTC (permalink / raw)
To: Aaro Koskinen; +Cc: devel, linux-mips, David Daney, richard, Greg Kroah-Hartman
On 09/30/2013 12:56 PM, Aaro Koskinen wrote:
> Hi,
>
> On Mon, Sep 30, 2013 at 12:41:59PM -0700, David Daney wrote:
>> On 09/30/2013 12:35 PM, Aaro Koskinen wrote:
>>> No, the original logic was already broken. The code assumed that the
>>> NAPI scheduled by the driver init gets executed always on CPU 0. The
>>> IRQ got enabled just because we are lucky.
>>
>> No. The default affinity for all irqs is CPU0 for just this reason.
>> So there was no luck involved.
>
> According the Kconfig, this driver can be compiled as a module:
>
>> config OCTEON_ETHERNET
>> tristate "Cavium Networks Octeon Ethernet support"
> [...]
>> To compile this driver as a module, choose M here. The module
>> will be called octeon-ethernet.
>
> What guarantees that CPU0 is around (or the smp_affinity is at its
> default value) by the time user executes modprobe?
Nothing enforced by the kernel. Just don't take CPU0 off-line and you
should be good to go.
There is a lot of room for improvement in the driver.
Really this whole thing of starting NAPI on multiple CPUs for the same
input queue is not good. It leads to loss of packet ordering when
forwarding.
What we really want is to have one POW group (input queue) per CPU and
only run one CPU per group. If we did that, we wouldn't have all these
affinity issues.
David Daney
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special
2013-09-30 21:08 ` David Daney
(?)
@ 2013-09-30 21:27 ` Aaro Koskinen
-1 siblings, 0 replies; 15+ messages in thread
From: Aaro Koskinen @ 2013-09-30 21:27 UTC (permalink / raw)
To: David Daney; +Cc: devel, linux-mips, David Daney, richard, Greg Kroah-Hartman
Hi,
On Mon, Sep 30, 2013 at 02:08:31PM -0700, David Daney wrote:
> On 09/30/2013 12:56 PM, Aaro Koskinen wrote:
> >What guarantees that CPU0 is around (or the smp_affinity is at its
> >default value) by the time user executes modprobe?
>
> Nothing enforced by the kernel. Just don't take CPU0 off-line and
> you should be good to go.
>
> There is a lot of room for improvement in the driver.
>
> Really this whole thing of starting NAPI on multiple CPUs for the
> same input queue is not good. It leads to loss of packet ordering
> when forwarding.
I agree. I have also another patch which deletes this functionality
altogether - should I post that one instead? In modern kernel you can use
RPS to steer packets to different cores/softirqs threads, so in a way the
driver currently duplicates some of the generic kernel functionality...
A.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special
2013-09-28 19:50 [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special Aaro Koskinen
` (3 preceding siblings ...)
2013-09-30 17:23 ` David Daney
@ 2013-10-03 20:36 ` Greg Kroah-Hartman
4 siblings, 0 replies; 15+ messages in thread
From: Greg Kroah-Hartman @ 2013-10-03 20:36 UTC (permalink / raw)
To: Aaro Koskinen; +Cc: devel, linux-mips, David Daney, richard
On Sat, Sep 28, 2013 at 10:50:33PM +0300, Aaro Koskinen wrote:
> Currently the driver assumes that CPU 0 is handling all the hard IRQs.
> This is wrong in Linux SMP systems where user is allowed to assign to
> hardware IRQs to any CPU. The driver will stop working if user sets
> smp_affinity so that interrupts end up being handled by other than CPU
> 0. The patch fixes that.
>
> Signed-off-by: Aaro Koskinen <aaro.koskinen@iki.fi>
> ---
> drivers/staging/octeon/ethernet-rx.c | 12 ++++++------
> 1 file changed, 6 insertions(+), 6 deletions(-)
Given the objections for this series, I've now dropped it from my queue.
If you want to resubmit it, please do so.
thanks,
greg k-h
^ permalink raw reply [flat|nested] 15+ messages in thread