[PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special

[PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special

[Aaro Koskinen]

Currently the driver assumes that CPU 0 is handling all the hard IRQs.
This is wrong in Linux SMP systems where user is allowed to assign to
hardware IRQs to any CPU. The driver will stop working if user sets
smp_affinity so that interrupts end up being handled by other than CPU
0. The patch fixes that.

Signed-off-by: Aaro Koskinen <aaro.koskinen@iki.fi>
---
 drivers/staging/octeon/ethernet-rx.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/drivers/staging/octeon/ethernet-rx.c b/drivers/staging/octeon/ethernet-rx.c
index e14a1bb..de831c1 100644
--- a/drivers/staging/octeon/ethernet-rx.c
+++ b/drivers/staging/octeon/ethernet-rx.c
@@ -80,6 +80,8 @@ struct cvm_oct_core_state {
 
 static struct cvm_oct_core_state core_state __cacheline_aligned_in_smp;
 
+static int cvm_irq_cpu = -1;
+
 static void cvm_oct_enable_napi(void *_)
 {
 	int cpu = smp_processor_id();
@@ -112,11 +114,7 @@ static void cvm_oct_no_more_work(void)
 {
 	int cpu = smp_processor_id();
 
-	/*
-	 * CPU zero is special.  It always has the irq enabled when
-	 * waiting for incoming packets.
-	 */
-	if (cpu == 0) {
+	if (cpu == cvm_irq_cpu) {
 		enable_irq(OCTEON_IRQ_WORKQ0 + pow_receive_group);
 		return;
 	}
@@ -135,6 +133,7 @@ static irqreturn_t cvm_oct_do_interrupt(int cpl, void *dev_id)
 {
 	/* Disable the IRQ and start napi_poll. */
 	disable_irq_nosync(OCTEON_IRQ_WORKQ0 + pow_receive_group);
+	cvm_irq_cpu = smp_processor_id();
 	cvm_oct_enable_napi(NULL);
 
 	return IRQ_HANDLED;
@@ -547,8 +546,9 @@ void cvm_oct_rx_initialize(void)
 	cvmx_write_csr(CVMX_POW_WQ_INT_PC, int_pc.u64);
 
 
-	/* Scheduld NAPI now.  This will indirectly enable interrupts. */
+	/* Schedule NAPI now. */
 	cvm_oct_enable_one_cpu();
+	enable_irq(OCTEON_IRQ_WORKQ0 + pow_receive_group);
 }
 
 void cvm_oct_rx_shutdown(void)
-- 
1.8.4.rc3

[PATCH 2/2] staging: octeon-ethernet: allow to use only 1 CPU for packet processing

[Aaro Koskinen]

Module parameter max_rx_cpus has off-by-one error. Fix that.

Signed-off-by: Aaro Koskinen <aaro.koskinen@iki.fi>
---
 drivers/staging/octeon/ethernet-rx.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/staging/octeon/ethernet-rx.c b/drivers/staging/octeon/ethernet-rx.c
index de831c1..fe1ee49 100644
--- a/drivers/staging/octeon/ethernet-rx.c
+++ b/drivers/staging/octeon/ethernet-rx.c
@@ -513,7 +513,7 @@ void cvm_oct_rx_initialize(void)
 	if (NULL == dev_for_napi)
 		panic("No net_devices were allocated.");
 
-	if (max_rx_cpus > 1  && max_rx_cpus < num_online_cpus())
+	if (max_rx_cpus >= 1 && max_rx_cpus < num_online_cpus())
 		atomic_set(&core_state.available_cores, max_rx_cpus);
 	else
 		atomic_set(&core_state.available_cores, num_online_cpus());
-- 
1.8.4.rc3

Re: [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special

[Richard Weinberger]

Am 28.09.2013 21:50, schrieb Aaro Koskinen:
> Currently the driver assumes that CPU 0 is handling all the hard IRQs.
> This is wrong in Linux SMP systems where user is allowed to assign to
> hardware IRQs to any CPU. The driver will stop working if user sets
> smp_affinity so that interrupts end up being handled by other than CPU
> 0. The patch fixes that.
> 
> Signed-off-by: Aaro Koskinen <aaro.koskinen@iki.fi>
> ---
>  drivers/staging/octeon/ethernet-rx.c | 12 ++++++------
>  1 file changed, 6 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/staging/octeon/ethernet-rx.c b/drivers/staging/octeon/ethernet-rx.c
> index e14a1bb..de831c1 100644
> --- a/drivers/staging/octeon/ethernet-rx.c
> +++ b/drivers/staging/octeon/ethernet-rx.c
> @@ -80,6 +80,8 @@ struct cvm_oct_core_state {
>  
>  static struct cvm_oct_core_state core_state __cacheline_aligned_in_smp;
>  
> +static int cvm_irq_cpu = -1;

Why are you introducing a new global variable here?
Can't you pass cvm_irq_cpu as argument to cvm_oct_enable_napi()?

Thanks,
//richard

Re: [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special

[Aaro Koskinen]

Hi,

On Sat, Sep 28, 2013 at 10:40:47PM +0200, Richard Weinberger wrote:
> Am 28.09.2013 21:50, schrieb Aaro Koskinen:
> > Currently the driver assumes that CPU 0 is handling all the hard IRQs.
> > This is wrong in Linux SMP systems where user is allowed to assign to
> > hardware IRQs to any CPU. The driver will stop working if user sets
> > smp_affinity so that interrupts end up being handled by other than CPU
> > 0. The patch fixes that.
> > 
> > Signed-off-by: Aaro Koskinen <aaro.koskinen@iki.fi>
> > ---
> >  drivers/staging/octeon/ethernet-rx.c | 12 ++++++------
> >  1 file changed, 6 insertions(+), 6 deletions(-)
> > 
> > diff --git a/drivers/staging/octeon/ethernet-rx.c b/drivers/staging/octeon/ethernet-rx.c
> > index e14a1bb..de831c1 100644
> > --- a/drivers/staging/octeon/ethernet-rx.c
> > +++ b/drivers/staging/octeon/ethernet-rx.c
> > @@ -80,6 +80,8 @@ struct cvm_oct_core_state {
> >  
> >  static struct cvm_oct_core_state core_state __cacheline_aligned_in_smp;
> >  
> > +static int cvm_irq_cpu = -1;
> 
> Why are you introducing a new global variable here?
> Can't you pass cvm_irq_cpu as argument to cvm_oct_enable_napi()?

This information needs to be accessed in cvm_oct_no_more_work().
Maybe I'm missing something obvious, but I don't get how the argument
could be accessed in or passed to napi poll routine which is calling
cvm_oct_no_more_work()?

A.

Re: [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special

[Richard Weinberger]

Am 28.09.2013 21:50, schrieb Aaro Koskinen:
> Currently the driver assumes that CPU 0 is handling all the hard IRQs.
> This is wrong in Linux SMP systems where user is allowed to assign to
> hardware IRQs to any CPU. The driver will stop working if user sets
> smp_affinity so that interrupts end up being handled by other than CPU
> 0. The patch fixes that.

You are right, sorry. I somehow mixed up the function names.

Thanks,
//richard

Re: [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special

[David Daney]

On 09/28/2013 12:50 PM, Aaro Koskinen wrote:
> Currently the driver assumes that CPU 0 is handling all the hard IRQs.
> This is wrong in Linux SMP systems where user is allowed to assign to
> hardware IRQs to any CPU. The driver will stop working if user sets
> smp_affinity so that interrupts end up being handled by other than CPU
> 0. The patch fixes that.
>
> Signed-off-by: Aaro Koskinen <aaro.koskinen@iki.fi>
> ---
>   drivers/staging/octeon/ethernet-rx.c | 12 ++++++------
>   1 file changed, 6 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/staging/octeon/ethernet-rx.c b/drivers/staging/octeon/ethernet-rx.c
> index e14a1bb..de831c1 100644
> --- a/drivers/staging/octeon/ethernet-rx.c
> +++ b/drivers/staging/octeon/ethernet-rx.c
> @@ -80,6 +80,8 @@ struct cvm_oct_core_state {
>
>   static struct cvm_oct_core_state core_state __cacheline_aligned_in_smp;
>
> +static int cvm_irq_cpu = -1;
> +
>   static void cvm_oct_enable_napi(void *_)
>   {
>   	int cpu = smp_processor_id();
> @@ -112,11 +114,7 @@ static void cvm_oct_no_more_work(void)
>   {
>   	int cpu = smp_processor_id();
>
> -	/*
> -	 * CPU zero is special.  It always has the irq enabled when
> -	 * waiting for incoming packets.
> -	 */
> -	if (cpu == 0) {
> +	if (cpu == cvm_irq_cpu) {
>   		enable_irq(OCTEON_IRQ_WORKQ0 + pow_receive_group);
>   		return;
>   	}
> @@ -135,6 +133,7 @@ static irqreturn_t cvm_oct_do_interrupt(int cpl, void *dev_id)
>   {
>   	/* Disable the IRQ and start napi_poll. */
>   	disable_irq_nosync(OCTEON_IRQ_WORKQ0 + pow_receive_group);
> +	cvm_irq_cpu = smp_processor_id();
>   	cvm_oct_enable_napi(NULL);
>
>   	return IRQ_HANDLED;
> @@ -547,8 +546,9 @@ void cvm_oct_rx_initialize(void)
>   	cvmx_write_csr(CVMX_POW_WQ_INT_PC, int_pc.u64);
>
>
> -	/* Scheduld NAPI now.  This will indirectly enable interrupts. */
> +	/* Schedule NAPI now. */
>   	cvm_oct_enable_one_cpu();
> +	enable_irq(OCTEON_IRQ_WORKQ0 + pow_receive_group);

The fact that you have to manually enable irqs here indicates that the 
patch is not good.

Either the enable_irq() is unnecessary, or you broke the logic for 
enabling NAPI on more than one CPU.

I am not sure which is the case, but I think it would be best if you 
supplied a fixed patch set that corrects whichever happens to be the case.

David Daney



>   }
>
>   void cvm_oct_rx_shutdown(void)
>

Re: [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special

[David Daney]

On 09/28/2013 12:50 PM, Aaro Koskinen wrote:
> Currently the driver assumes that CPU 0 is handling all the hard IRQs.
> This is wrong in Linux SMP systems where user is allowed to assign to
> hardware IRQs to any CPU. The driver will stop working if user sets
> smp_affinity so that interrupts end up being handled by other than CPU
> 0. The patch fixes that.
>
> Signed-off-by: Aaro Koskinen <aaro.koskinen@iki.fi>
> ---
>   drivers/staging/octeon/ethernet-rx.c | 12 ++++++------
>   1 file changed, 6 insertions(+), 6 deletions(-)
>
> diff --git a/drivers/staging/octeon/ethernet-rx.c b/drivers/staging/octeon/ethernet-rx.c
> index e14a1bb..de831c1 100644
> --- a/drivers/staging/octeon/ethernet-rx.c
> +++ b/drivers/staging/octeon/ethernet-rx.c
> @@ -80,6 +80,8 @@ struct cvm_oct_core_state {
>
>   static struct cvm_oct_core_state core_state __cacheline_aligned_in_smp;
>
> +static int cvm_irq_cpu = -1;
> +
>   static void cvm_oct_enable_napi(void *_)
>   {
>   	int cpu = smp_processor_id();
> @@ -112,11 +114,7 @@ static void cvm_oct_no_more_work(void)
>   {
>   	int cpu = smp_processor_id();
>
> -	/*
> -	 * CPU zero is special.  It always has the irq enabled when
> -	 * waiting for incoming packets.
> -	 */
> -	if (cpu == 0) {
> +	if (cpu == cvm_irq_cpu) {
>   		enable_irq(OCTEON_IRQ_WORKQ0 + pow_receive_group);
>   		return;
>   	}
> @@ -135,6 +133,7 @@ static irqreturn_t cvm_oct_do_interrupt(int cpl, void *dev_id)
>   {
>   	/* Disable the IRQ and start napi_poll. */
>   	disable_irq_nosync(OCTEON_IRQ_WORKQ0 + pow_receive_group);
> +	cvm_irq_cpu = smp_processor_id();
>   	cvm_oct_enable_napi(NULL);
>
>   	return IRQ_HANDLED;
> @@ -547,8 +546,9 @@ void cvm_oct_rx_initialize(void)
>   	cvmx_write_csr(CVMX_POW_WQ_INT_PC, int_pc.u64);
>
>
> -	/* Scheduld NAPI now.  This will indirectly enable interrupts. */
> +	/* Schedule NAPI now. */
>   	cvm_oct_enable_one_cpu();
> +	enable_irq(OCTEON_IRQ_WORKQ0 + pow_receive_group);

The fact that you have to manually enable irqs here indicates that the 
patch is not good.

Either the enable_irq() is unnecessary, or you broke the logic for 
enabling NAPI on more than one CPU.

I am not sure which is the case, but I think it would be best if you 
supplied a fixed patch set that corrects whichever happens to be the case.

David Daney



>   }
>
>   void cvm_oct_rx_shutdown(void)
>

Re: [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special

[Aaro Koskinen]

Hi,

On Mon, Sep 30, 2013 at 10:23:10AM -0700, David Daney wrote:
> On 09/28/2013 12:50 PM, Aaro Koskinen wrote:
> >Currently the driver assumes that CPU 0 is handling all the hard IRQs.
> >This is wrong in Linux SMP systems where user is allowed to assign to
> >hardware IRQs to any CPU. The driver will stop working if user sets
> >smp_affinity so that interrupts end up being handled by other than CPU
> >0. The patch fixes that.
> >
> >Signed-off-by: Aaro Koskinen <aaro.koskinen@iki.fi>
> >---
> >  drivers/staging/octeon/ethernet-rx.c | 12 ++++++------
> >  1 file changed, 6 insertions(+), 6 deletions(-)
> >
> >diff --git a/drivers/staging/octeon/ethernet-rx.c b/drivers/staging/octeon/ethernet-rx.c
> >index e14a1bb..de831c1 100644
> >--- a/drivers/staging/octeon/ethernet-rx.c
> >+++ b/drivers/staging/octeon/ethernet-rx.c
> >@@ -80,6 +80,8 @@ struct cvm_oct_core_state {
> >
> >  static struct cvm_oct_core_state core_state __cacheline_aligned_in_smp;
> >
> >+static int cvm_irq_cpu = -1;
> >+
> >  static void cvm_oct_enable_napi(void *_)
> >  {
> >  	int cpu = smp_processor_id();
> >@@ -112,11 +114,7 @@ static void cvm_oct_no_more_work(void)
> >  {
> >  	int cpu = smp_processor_id();
> >
> >-	/*
> >-	 * CPU zero is special.  It always has the irq enabled when
> >-	 * waiting for incoming packets.
> >-	 */
> >-	if (cpu == 0) {
> >+	if (cpu == cvm_irq_cpu) {
> >  		enable_irq(OCTEON_IRQ_WORKQ0 + pow_receive_group);
> >  		return;
> >  	}
> >@@ -135,6 +133,7 @@ static irqreturn_t cvm_oct_do_interrupt(int cpl, void *dev_id)
> >  {
> >  	/* Disable the IRQ and start napi_poll. */
> >  	disable_irq_nosync(OCTEON_IRQ_WORKQ0 + pow_receive_group);
> >+	cvm_irq_cpu = smp_processor_id();
> >  	cvm_oct_enable_napi(NULL);
> >
> >  	return IRQ_HANDLED;
> >@@ -547,8 +546,9 @@ void cvm_oct_rx_initialize(void)
> >  	cvmx_write_csr(CVMX_POW_WQ_INT_PC, int_pc.u64);
> >
> >
> >-	/* Scheduld NAPI now.  This will indirectly enable interrupts. */
> >+	/* Schedule NAPI now. */
> >  	cvm_oct_enable_one_cpu();
> >+	enable_irq(OCTEON_IRQ_WORKQ0 + pow_receive_group);
> 
> The fact that you have to manually enable irqs here indicates that
> the patch is not good.
> 
> Either the enable_irq() is unnecessary, or you broke the logic for
> enabling NAPI on more than one CPU.
> 
> I am not sure which is the case, but I think it would be best if you
> supplied a fixed patch set that corrects whichever happens to be the
> case.

No, the original logic was already broken. The code assumed that the
NAPI scheduled by the driver init gets executed always on CPU 0. The
IRQ got enabled just because we are lucky.

The patch removes such assumption. During the driver init, the IRQ is
disabled and cvm_irq_cpu is -1. So when the NAPI scheduled (on whatever
CPU) by the init is done, the check in cvm_oct_no_more_work() will be
false and the IRQ remains disabled. So the init routine has to enable IRQ
"manually". This is a special case only for the driver initialization.

During the normal operation, the IRQ handler record the CPU (on which
it's going the schedule for the first NAPI poll) in cvm_irq_cpu so
cvm_oct_no_more_work() will always re-enable the IRQ correctly.

A.

Re: [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special

[David Daney]

On 09/30/2013 12:35 PM, Aaro Koskinen wrote:
> Hi,
>
> On Mon, Sep 30, 2013 at 10:23:10AM -0700, David Daney wrote:
>> On 09/28/2013 12:50 PM, Aaro Koskinen wrote:
>>> Currently the driver assumes that CPU 0 is handling all the hard IRQs.
>>> This is wrong in Linux SMP systems where user is allowed to assign to
>>> hardware IRQs to any CPU. The driver will stop working if user sets
>>> smp_affinity so that interrupts end up being handled by other than CPU
>>> 0. The patch fixes that.
>>>
>>> Signed-off-by: Aaro Koskinen <aaro.koskinen@iki.fi>
>>> ---
>>>   drivers/staging/octeon/ethernet-rx.c | 12 ++++++------
>>>   1 file changed, 6 insertions(+), 6 deletions(-)
>>>
>>> diff --git a/drivers/staging/octeon/ethernet-rx.c b/drivers/staging/octeon/ethernet-rx.c
>>> index e14a1bb..de831c1 100644
>>> --- a/drivers/staging/octeon/ethernet-rx.c
>>> +++ b/drivers/staging/octeon/ethernet-rx.c
>>> @@ -80,6 +80,8 @@ struct cvm_oct_core_state {
>>>
>>>   static struct cvm_oct_core_state core_state __cacheline_aligned_in_smp;
>>>
>>> +static int cvm_irq_cpu = -1;
>>> +
>>>   static void cvm_oct_enable_napi(void *_)
>>>   {
>>>   	int cpu = smp_processor_id();
>>> @@ -112,11 +114,7 @@ static void cvm_oct_no_more_work(void)
>>>   {
>>>   	int cpu = smp_processor_id();
>>>
>>> -	/*
>>> -	 * CPU zero is special.  It always has the irq enabled when
>>> -	 * waiting for incoming packets.
>>> -	 */
>>> -	if (cpu == 0) {
>>> +	if (cpu == cvm_irq_cpu) {
>>>   		enable_irq(OCTEON_IRQ_WORKQ0 + pow_receive_group);
>>>   		return;
>>>   	}
>>> @@ -135,6 +133,7 @@ static irqreturn_t cvm_oct_do_interrupt(int cpl, void *dev_id)
>>>   {
>>>   	/* Disable the IRQ and start napi_poll. */
>>>   	disable_irq_nosync(OCTEON_IRQ_WORKQ0 + pow_receive_group);
>>> +	cvm_irq_cpu = smp_processor_id();
>>>   	cvm_oct_enable_napi(NULL);
>>>
>>>   	return IRQ_HANDLED;
>>> @@ -547,8 +546,9 @@ void cvm_oct_rx_initialize(void)
>>>   	cvmx_write_csr(CVMX_POW_WQ_INT_PC, int_pc.u64);
>>>
>>>
>>> -	/* Scheduld NAPI now.  This will indirectly enable interrupts. */
>>> +	/* Schedule NAPI now. */
>>>   	cvm_oct_enable_one_cpu();
>>> +	enable_irq(OCTEON_IRQ_WORKQ0 + pow_receive_group);
>>
>> The fact that you have to manually enable irqs here indicates that
>> the patch is not good.
>>
>> Either the enable_irq() is unnecessary, or you broke the logic for
>> enabling NAPI on more than one CPU.
>>
>> I am not sure which is the case, but I think it would be best if you
>> supplied a fixed patch set that corrects whichever happens to be the
>> case.
>
> No, the original logic was already broken. The code assumed that the
> NAPI scheduled by the driver init gets executed always on CPU 0. The
> IRQ got enabled just because we are lucky.

No.  The default affinity for all irqs is CPU0 for just this reason.  So 
there was no luck involved.

>
> The patch removes such assumption. During the driver init, the IRQ is
> disabled and cvm_irq_cpu is -1. So when the NAPI scheduled (on whatever
> CPU) by the init is done, the check in cvm_oct_no_more_work() will be
> false and the IRQ remains disabled. So the init routine has to enable IRQ
> "manually". This is a special case only for the driver initialization.

No.  We schedule NAPI at this point.  If there are no packets available, 
the normal driver logic enables the irq.  There is no need to explicitly 
enable it in cvm_oct_rx_initialize().

If this doesn't work, and you need to have an additional place that you 
enable the irq, then you broke something.


>
> During the normal operation, the IRQ handler record the CPU (on which
> it's going the schedule for the first NAPI poll) in cvm_irq_cpu so
> cvm_oct_no_more_work() will always re-enable the IRQ correctly.
>
> A.
>
>
>

Re: [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special

[David Daney]

On 09/30/2013 12:35 PM, Aaro Koskinen wrote:
> Hi,
>
> On Mon, Sep 30, 2013 at 10:23:10AM -0700, David Daney wrote:
>> On 09/28/2013 12:50 PM, Aaro Koskinen wrote:
>>> Currently the driver assumes that CPU 0 is handling all the hard IRQs.
>>> This is wrong in Linux SMP systems where user is allowed to assign to
>>> hardware IRQs to any CPU. The driver will stop working if user sets
>>> smp_affinity so that interrupts end up being handled by other than CPU
>>> 0. The patch fixes that.
>>>
>>> Signed-off-by: Aaro Koskinen <aaro.koskinen@iki.fi>
>>> ---
>>>   drivers/staging/octeon/ethernet-rx.c | 12 ++++++------
>>>   1 file changed, 6 insertions(+), 6 deletions(-)
>>>
>>> diff --git a/drivers/staging/octeon/ethernet-rx.c b/drivers/staging/octeon/ethernet-rx.c
>>> index e14a1bb..de831c1 100644
>>> --- a/drivers/staging/octeon/ethernet-rx.c
>>> +++ b/drivers/staging/octeon/ethernet-rx.c
>>> @@ -80,6 +80,8 @@ struct cvm_oct_core_state {
>>>
>>>   static struct cvm_oct_core_state core_state __cacheline_aligned_in_smp;
>>>
>>> +static int cvm_irq_cpu = -1;
>>> +
>>>   static void cvm_oct_enable_napi(void *_)
>>>   {
>>>   	int cpu = smp_processor_id();
>>> @@ -112,11 +114,7 @@ static void cvm_oct_no_more_work(void)
>>>   {
>>>   	int cpu = smp_processor_id();
>>>
>>> -	/*
>>> -	 * CPU zero is special.  It always has the irq enabled when
>>> -	 * waiting for incoming packets.
>>> -	 */
>>> -	if (cpu == 0) {
>>> +	if (cpu == cvm_irq_cpu) {
>>>   		enable_irq(OCTEON_IRQ_WORKQ0 + pow_receive_group);
>>>   		return;
>>>   	}
>>> @@ -135,6 +133,7 @@ static irqreturn_t cvm_oct_do_interrupt(int cpl, void *dev_id)
>>>   {
>>>   	/* Disable the IRQ and start napi_poll. */
>>>   	disable_irq_nosync(OCTEON_IRQ_WORKQ0 + pow_receive_group);
>>> +	cvm_irq_cpu = smp_processor_id();
>>>   	cvm_oct_enable_napi(NULL);
>>>
>>>   	return IRQ_HANDLED;
>>> @@ -547,8 +546,9 @@ void cvm_oct_rx_initialize(void)
>>>   	cvmx_write_csr(CVMX_POW_WQ_INT_PC, int_pc.u64);
>>>
>>>
>>> -	/* Scheduld NAPI now.  This will indirectly enable interrupts. */
>>> +	/* Schedule NAPI now. */
>>>   	cvm_oct_enable_one_cpu();
>>> +	enable_irq(OCTEON_IRQ_WORKQ0 + pow_receive_group);
>>
>> The fact that you have to manually enable irqs here indicates that
>> the patch is not good.
>>
>> Either the enable_irq() is unnecessary, or you broke the logic for
>> enabling NAPI on more than one CPU.
>>
>> I am not sure which is the case, but I think it would be best if you
>> supplied a fixed patch set that corrects whichever happens to be the
>> case.
>
> No, the original logic was already broken. The code assumed that the
> NAPI scheduled by the driver init gets executed always on CPU 0. The
> IRQ got enabled just because we are lucky.

No.  The default affinity for all irqs is CPU0 for just this reason.  So 
there was no luck involved.

>
> The patch removes such assumption. During the driver init, the IRQ is
> disabled and cvm_irq_cpu is -1. So when the NAPI scheduled (on whatever
> CPU) by the init is done, the check in cvm_oct_no_more_work() will be
> false and the IRQ remains disabled. So the init routine has to enable IRQ
> "manually". This is a special case only for the driver initialization.

No.  We schedule NAPI at this point.  If there are no packets available, 
the normal driver logic enables the irq.  There is no need to explicitly 
enable it in cvm_oct_rx_initialize().

If this doesn't work, and you need to have an additional place that you 
enable the irq, then you broke something.


>
> During the normal operation, the IRQ handler record the CPU (on which
> it's going the schedule for the first NAPI poll) in cvm_irq_cpu so
> cvm_oct_no_more_work() will always re-enable the IRQ correctly.
>
> A.
>
>
>

Re: [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special

[Aaro Koskinen]

Hi,

On Mon, Sep 30, 2013 at 12:41:59PM -0700, David Daney wrote:
> On 09/30/2013 12:35 PM, Aaro Koskinen wrote:
> >No, the original logic was already broken. The code assumed that the
> >NAPI scheduled by the driver init gets executed always on CPU 0. The
> >IRQ got enabled just because we are lucky.
> 
> No.  The default affinity for all irqs is CPU0 for just this reason.
> So there was no luck involved.

According the Kconfig, this driver can be compiled as a module:

> config OCTEON_ETHERNET
>	tristate "Cavium Networks Octeon Ethernet support"
[...]
>	To compile this driver as a module, choose M here.  The module
>	will be called octeon-ethernet.

What guarantees that CPU0 is around (or the smp_affinity is at its
default value) by the time user executes modprobe?

A.

Re: [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special

[David Daney]

On 09/30/2013 12:56 PM, Aaro Koskinen wrote:
> Hi,
>
> On Mon, Sep 30, 2013 at 12:41:59PM -0700, David Daney wrote:
>> On 09/30/2013 12:35 PM, Aaro Koskinen wrote:
>>> No, the original logic was already broken. The code assumed that the
>>> NAPI scheduled by the driver init gets executed always on CPU 0. The
>>> IRQ got enabled just because we are lucky.
>>
>> No.  The default affinity for all irqs is CPU0 for just this reason.
>> So there was no luck involved.
>
> According the Kconfig, this driver can be compiled as a module:
>
>> config OCTEON_ETHERNET
>> 	tristate "Cavium Networks Octeon Ethernet support"
> [...]
>> 	To compile this driver as a module, choose M here.  The module
>> 	will be called octeon-ethernet.
>
> What guarantees that CPU0 is around (or the smp_affinity is at its
> default value) by the time user executes modprobe?

Nothing enforced by the kernel.  Just don't take CPU0 off-line and you 
should be good to go.

There is a lot of room for improvement in the driver.

Really this whole thing of starting NAPI on multiple CPUs for the same 
input queue is not good.  It leads to loss of packet ordering when 
forwarding.

What we really want is to have one POW group (input queue) per CPU and 
only run one CPU per group.  If we did that, we wouldn't have all these 
affinity issues.

David Daney

Re: [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special

[David Daney]

On 09/30/2013 12:56 PM, Aaro Koskinen wrote:
> Hi,
>
> On Mon, Sep 30, 2013 at 12:41:59PM -0700, David Daney wrote:
>> On 09/30/2013 12:35 PM, Aaro Koskinen wrote:
>>> No, the original logic was already broken. The code assumed that the
>>> NAPI scheduled by the driver init gets executed always on CPU 0. The
>>> IRQ got enabled just because we are lucky.
>>
>> No.  The default affinity for all irqs is CPU0 for just this reason.
>> So there was no luck involved.
>
> According the Kconfig, this driver can be compiled as a module:
>
>> config OCTEON_ETHERNET
>> 	tristate "Cavium Networks Octeon Ethernet support"
> [...]
>> 	To compile this driver as a module, choose M here.  The module
>> 	will be called octeon-ethernet.
>
> What guarantees that CPU0 is around (or the smp_affinity is at its
> default value) by the time user executes modprobe?

Nothing enforced by the kernel.  Just don't take CPU0 off-line and you 
should be good to go.

There is a lot of room for improvement in the driver.

Really this whole thing of starting NAPI on multiple CPUs for the same 
input queue is not good.  It leads to loss of packet ordering when 
forwarding.

What we really want is to have one POW group (input queue) per CPU and 
only run one CPU per group.  If we did that, we wouldn't have all these 
affinity issues.

David Daney

Re: [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special

[Aaro Koskinen]

Hi,

On Mon, Sep 30, 2013 at 02:08:31PM -0700, David Daney wrote:
> On 09/30/2013 12:56 PM, Aaro Koskinen wrote:
> >What guarantees that CPU0 is around (or the smp_affinity is at its
> >default value) by the time user executes modprobe?
> 
> Nothing enforced by the kernel.  Just don't take CPU0 off-line and
> you should be good to go.
> 
> There is a lot of room for improvement in the driver.
> 
> Really this whole thing of starting NAPI on multiple CPUs for the
> same input queue is not good.  It leads to loss of packet ordering
> when forwarding.

I agree. I have also another patch which deletes this functionality
altogether - should I post that one instead? In modern kernel you can use
RPS to steer packets to different cores/softirqs threads, so in a way the
driver currently duplicates some of the generic kernel functionality...

A.

Re: [PATCH 1/2] staging: octeon-ethernet: don't assume that CPU 0 is special

[Greg Kroah-Hartman]

On Sat, Sep 28, 2013 at 10:50:33PM +0300, Aaro Koskinen wrote:
> Currently the driver assumes that CPU 0 is handling all the hard IRQs.
> This is wrong in Linux SMP systems where user is allowed to assign to
> hardware IRQs to any CPU. The driver will stop working if user sets
> smp_affinity so that interrupts end up being handled by other than CPU
> 0. The patch fixes that.
> 
> Signed-off-by: Aaro Koskinen <aaro.koskinen@iki.fi>
> ---
>  drivers/staging/octeon/ethernet-rx.c | 12 ++++++------
>  1 file changed, 6 insertions(+), 6 deletions(-)

Given the objections for this series, I've now dropped it from my queue.
If you want to resubmit it, please do so.

thanks,

greg k-h