[Xenomai-core] Compile-time bug, and problem with PPC440 ethernet

[Xenomai-core] Compile-time bug, and problem with PPC440 ethernet

[Steven A. Falco]

I am building kernel 2.6.25.4 from DENX with Xenomai 2.4.4 for PPC440EPx
(sequoia development board).

The kernel tries to use DHCP to obtain network settings.  With IPIPE
disabled, this works perfectly.  However, when I enable IPIPE, the board
sends packets ok, but does not receive packets - I can see the DHCP with
a sniffer, but the development board does not receive them.

Also, in order to disable IPIPE, I had to patch
kernel/time/tick-common.c around line 168.  It is missing an ifdef:

diff --git a/kernel/time/tick-common.c b/kernel/time/tick-common.c
index 58bfacf..3a735b8 100644
--- a/kernel/time/tick-common.c
+++ b/kernel/time/tick-common.c
@@ -163,9 +163,11 @@ static void tick_setup_device(struct tick_device *td,

        td->evtdev = newdev;

+#ifdef CONFIG_IPIPE
        /* I-pipe: derive global tick IRQ from CPU 0 */
        if (cpu == 0)
                ipipe_update_tick_evtdev(newdev);
+#endif

        /*
         * When the device is not per cpu, pin the interrupt to the

Signed-off-by: Steve Falco <sfalco@domain.hid>

If anyone has suggestions as to why IPIPE is blocking received ethernet
packets, I'd appreciate it.

    Steve

Re: [Xenomai-core] Compile-time bug, and problem with PPC440 ethernet

[Steven A. Falco]

A bit more info: Interrupt numbers are 0x15 (txeob) and 0x16 (rxeob). 
I've added some printk, and at the point where the DHCP message is sent
out, I get:

DHCP: sending class identifier "hydra_temp"
stalled stage                     <- printed in __ipipe_walk_pipeline
(see below)
__ipipe_grab_irq 15               <- printed at the beginning of
__ipipe_grab_irq
ready to walk 15                  <- printed at the finalize:, before
__ipipe_walk_pipeline is called
stalled stage                     <- printed in __ipipe_walk_pipeline
(see below)
__ipipe_grab_irq 16               <- printed at the beginning of
__ipipe_grab_irq
ready to walk 16                  <- printed at the finalize:, before
__ipipe_walk_pipeline is called
stalled stage                     <- see below
stalled stage                     <- etc
ipipe_cpudom_var != 0            
next_domain == this_domain
stalled stage

So, while I don't pretend to understand this yet, it looks like the
interrupts are coming into ipipe, but perhaps not getting delivered for
some reason?

    Steve


void __ipipe_walk_pipeline(struct list_head *pos)
{
    struct ipipe_domain *this_domain = ipipe_current_domain, *next_domain;

    while (pos != &__ipipe_pipeline) {

        next_domain = list_entry(pos, struct ipipe_domain, p_link);

        if (test_bit(IPIPE_STALL_FLAG, &ipipe_cpudom_var(next_domain,
status))) {
            printk("stalled stage\n");
            break;    /* Stalled stage -- do not go further. */
        }

        if (ipipe_cpudom_var(next_domain, irqpend_himask) != 0) {
            printk("ipipe_cpudom_var != 0\n");

            if (next_domain == this_domain) {
                printk("next_domain == this_domain\n");
                __ipipe_sync_pipeline(IPIPE_IRQMASK_ANY);
            } else {

                printk("next_domain != this_domain\n");
                ipipe_cpudom_var(this_domain, evsync) = 0;
                ipipe_current_domain = next_domain;
                ipipe_suspend_domain();    /* Sync stage and propagate
interrupts. */

                if (ipipe_current_domain == next_domain)
                    ipipe_current_domain = this_domain;
                /*
                 * Otherwise, something changed the current domain under our
                 * feet recycling the register set; do not override the new
                 * domain.
                 */

                if (ipipe_cpudom_var(this_domain, irqpend_himask) != 0 &&
                    !test_bit(IPIPE_STALL_FLAG,
                          &ipipe_cpudom_var(this_domain, status))) {
                    printk("recycling the register set\n");
                    __ipipe_sync_pipeline(IPIPE_IRQMASK_ANY);
                }
            }

            break;
        } else if (next_domain == this_domain) {
            printk("ipipe_cpudom_var == 0\n");
            printk("next_domain == this_domain\n");
            break;
        }

        pos = next_domain->p_link.next;
    }
}

Re: [Xenomai-core] Compile-time bug, and problem with PPC440 ethernet

[Philippe Gerum]

Steven A. Falco wrote:
> I am building kernel 2.6.25.4 from DENX with Xenomai 2.4.4 for PPC440EPx
> (sequoia development board).
> 
> The kernel tries to use DHCP to obtain network settings.  With IPIPE
> disabled, this works perfectly.  However, when I enable IPIPE, the board
> sends packets ok, but does not receive packets - I can see the DHCP with
> a sniffer, but the development board does not receive them.
>

External interrupts are probably locked out by the pipeline engine; I've fixed a
similar issue in recent patches for other PICs. Which I-pipe patch release are
you using?

> Also, in order to disable IPIPE, I had to patch
> kernel/time/tick-common.c around line 168.  It is missing an ifdef:
> 
> diff --git a/kernel/time/tick-common.c b/kernel/time/tick-common.c
> index 58bfacf..3a735b8 100644
> --- a/kernel/time/tick-common.c
> +++ b/kernel/time/tick-common.c
> @@ -163,9 +163,11 @@ static void tick_setup_device(struct tick_device *td,
> 
>         td->evtdev = newdev;
> 
> +#ifdef CONFIG_IPIPE
>         /* I-pipe: derive global tick IRQ from CPU 0 */
>         if (cpu == 0)
>                 ipipe_update_tick_evtdev(newdev);
> +#endif
> 
>         /*
>          * When the device is not per cpu, pin the interrupt to the
> 
> Signed-off-by: Steve Falco <sfalco@domain.hid>
> 
> If anyone has suggestions as to why IPIPE is blocking received ethernet
> packets, I'd appreciate it.
> 
>     Steve
> 
> 
> _______________________________________________
> Xenomai-core mailing list
> Xenomai-core@domain.hid
> https://mail.gna.org/listinfo/xenomai-core
> 


-- 
Philippe.

Re: [Xenomai-core] Compile-time bug, and problem with PPC440 ethernet

[Steven A. Falco]

[-- Attachment #1: Type: text/plain, Size: 1540 bytes --]

Philippe Gerum wrote:
> Steven A. Falco wrote:
>   
>> I am building kernel 2.6.25.4 from DENX with Xenomai 2.4.4 for PPC440EPx
>> (sequoia development board).
>>
>> The kernel tries to use DHCP to obtain network settings.  With IPIPE
>> disabled, this works perfectly.  However, when I enable IPIPE, the board
>> sends packets ok, but does not receive packets - I can see the DHCP with
>> a sniffer, but the development board does not receive them.
>
>   
> External interrupts are probably locked out by the pipeline engine;
> I've fixed a
> similar issue in recent patches for other PICs. Which I-pipe patch
> release are
> you using?


I am using the patch that came with Xenomai 2.4.4, namely:
adeos-ipipe-2.6.25-powerpc-DENX-2.2-02.patch

>> Also, in order to disable IPIPE, I had to patch
>> kernel/time/tick-common.c around line 168. It is missing an ifdef:
>>
>> diff --git a/kernel/time/tick-common.c b/kernel/time/tick-common.c
>> index 58bfacf..3a735b8 100644
>> --- a/kernel/time/tick-common.c
>> +++ b/kernel/time/tick-common.c
>> @@ -163,9 +163,11 @@ static void tick_setup_device(struct tick_device
>> *td,
>>
>> td->evtdev = newdev;
>>
>> +#ifdef CONFIG_IPIPE
>> /* I-pipe: derive global tick IRQ from CPU 0 */
>> if (cpu == 0)
>> ipipe_update_tick_evtdev(newdev);
>> +#endif
>>
>> /*
>> * When the device is not per cpu, pin the interrupt to the
>>
>> Signed-off-by: Steve Falco <sfalco@domain.hid>
>>
>> If anyone has suggestions as to why IPIPE is blocking received ethernet
>> packets, I'd appreciate it.
>>
>> Steve



[-- Attachment #2: Type: text/html, Size: 2256 bytes --]

Re: [Xenomai-core] Compile-time bug, and problem with PPC440 ethernet

[Philippe Gerum]

Steven A. Falco wrote:
> Philippe Gerum wrote:
>> Steven A. Falco wrote:
>>   
>>> I am building kernel 2.6.25.4 from DENX with Xenomai 2.4.4 for PPC440EPx
>>> (sequoia development board).
>>>
>>> The kernel tries to use DHCP to obtain network settings.  With IPIPE
>>> disabled, this works perfectly.  However, when I enable IPIPE, the board
>>> sends packets ok, but does not receive packets - I can see the DHCP with
>>> a sniffer, but the development board does not receive them.
>>
>>   
>> External interrupts are probably locked out by the pipeline engine;
>> I've fixed a
>> similar issue in recent patches for other PICs. Which I-pipe patch
>> release are
>> you using?
> 
> 
> I am using the patch that came with Xenomai 2.4.4, namely:
> adeos-ipipe-2.6.25-powerpc-DENX-2.2-02.patch
> 

I can't reproduce this issue on a 440EP board, but I still have to to put my
hands on a 440EPX to check this. However, both should be using the common 44x
PIC support, so I don't expect big changes here.

Are you using the powerpc/ branch, or legacy ppc/ one for building the sequoia
kernel?

-- 
Philippe.

Re: [Xenomai-core] Compile-time bug, and problem with PPC440 ethernet

[Philippe Gerum]

Steven A. Falco wrote:
> 
>>> I am using the patch that came with Xenomai 2.4.4, namely:
>>> adeos-ipipe-2.6.25-powerpc-DENX-2.2-02.patch
>>>
>>>     
>>
>> I can't reproduce this issue on a 440EP board, but I still have to to put my
>> hands on a 440EPX to check this. However, both should be using the common 44x
>> PIC support, so I don't expect big changes here.
>>
>> Are you using the powerpc/ branch, or legacy ppc/ one for building the sequoia
>> kernel?
>>   
> 
> ARCH=powerpc.

Ah. So it's the UIC support that likely breaks. Could you try this patch?

diff --git a/arch/powerpc/sysdev/uic.c b/arch/powerpc/sysdev/uic.c
index dd40f64..4eff1ab 100644
--- a/arch/powerpc/sysdev/uic.c
+++ b/arch/powerpc/sysdev/uic.c
@@ -111,7 +111,6 @@ static void uic_mask_ack_irq(unsigned int virq)

 	sr = 1 << (31-src);
 	spin_lock_irqsave(&uic->lock, flags);
-	ipipe_irq_lock(virq);
 	er = mfdcr(uic->dcrbase + UIC_ER);
 	er &= ~sr;
 	mtdcr(uic->dcrbase + UIC_ER, er);

In case it is not enough, try removing all the ipipe_irq_lock/unlock calls from
arch/powerpc/sysdev/uic.c. If that works eventually, I'll sort the mess out later.

  I started with the DENX-v2.6.25-stable branch from the
> DENX linux-2.6-denx git tree, then applied Xenomai-2.4.4 using its
> install script.  The toolchain is eldk-4.2 for ppc_4xxFP.
> 
> Is there some debugging I can turn on to help you track this down?  I
> can also add additional printk statements.
> 
> In a case of "bad timing", I will be going on vacation tomorrow, so I
> will do what I can to gather more data today.  Otherwise, I'll have to
> pick this up next Thursday (26th). :-(
> 
>     Thanks,
>     Steve
> 


-- 
Philippe.

Re: [Xenomai-core] Compile-time bug, and problem with PPC440 ethernet

[Philippe Gerum]

Steven A. Falco wrote:
> Philippe Gerum wrote:
>> Steven A. Falco wrote:
>>   
>>>>> I am using the patch that came with Xenomai 2.4.4, namely:
>>>>> adeos-ipipe-2.6.25-powerpc-DENX-2.2-02.patch
>>>>>
>>>>>     
>>>>>         
>>>> I can't reproduce this issue on a 440EP board, but I still have to to put my
>>>> hands on a 440EPX to check this. However, both should be using the common 44x
>>>> PIC support, so I don't expect big changes here.
>>>>
>>>> Are you using the powerpc/ branch, or legacy ppc/ one for building the sequoia
>>>> kernel?
>>>>   
>>>>       
>>> ARCH=powerpc.
>>>     
>>
>> Ah. So it's the UIC support that likely breaks. Could you try this patch?
>>
>> diff --git a/arch/powerpc/sysdev/uic.c b/arch/powerpc/sysdev/uic.c
>> index dd40f64..4eff1ab 100644
>> --- a/arch/powerpc/sysdev/uic.c
>> +++ b/arch/powerpc/sysdev/uic.c
>> @@ -111,7 +111,6 @@ static void uic_mask_ack_irq(unsigned int virq)
>>
>>  	sr = 1 << (31-src);
>>  	spin_lock_irqsave(&uic->lock, flags);
>> -	ipipe_irq_lock(virq);
>>  	er = mfdcr(uic->dcrbase + UIC_ER);
>>  	er &= ~sr;
>>  	mtdcr(uic->dcrbase + UIC_ER, er);
>>
>> In case it is not enough, try removing all the ipipe_irq_lock/unlock calls from
>> arch/powerpc/sysdev/uic.c. If that works eventually, I'll sort the mess out later.
>>   
> 
> Beautiful.  The patch works!  I now get DHCP replies.
> 
> Is this the patch you will put into the official tree, or do you still
> need to do more?
>

One thing, could you confirm that your network card relies on edge interrupts
(and not level)?

Aside of that, it should be ok. It's the same IRQ lock out issue fixed in the
ppc/ branch recently.

> Also, please include my compile-time patch, if that is acceptible.
> 

I recently committed a different fix for the same issue after you reported it.
This should work without requiring additional #ifdef'ing as well. Thanks for the
heads up.

--- a/include/asm-powerpc/ipipe.h
+++ b/include/asm-powerpc/ipipe.h
@@ -62,8 +62,6 @@
 		local_irq_enable_hw(); x;				\
 	} )

-#define ipipe_update_tick_evtdev(evtdev)	do { } while (0)
-
 struct ipipe_domain;

 struct ipipe_sysinfo {
@@ -209,4 +207,6 @@ do {									\

 #endif /* CONFIG_IPIPE */

+#define ipipe_update_tick_evtdev(evtdev)	do { } while (0)
+
 #endif /* !__ASM_POWERPC_IPIPE_H */


-- 
Philippe.

Re: [Xenomai-core] Compile-time bug, and problem with PPC440 ethernet

[Steven A. Falco]

[-- Attachment #1: Type: text/plain, Size: 1764 bytes --]


>> Beautiful.  The patch works!  I now get DHCP replies.
>>
>> Is this the patch you will put into the official tree, or do you still
>> need to do more?
>>
>>     
>
> One thing, could you confirm that your network card relies on edge interrupts
> (and not level)?
>
> Aside of that, it should be ok. It's the same IRQ lock out issue fixed in the
> ppc/ branch recently.
>
>   

Sorry for the delay in getting back to you - vacation. 

The ethernet interrupts on the PPC440 are level-sensitive.  I gather
that is not what you were expecting, so I wonder what the implications are.

After looking at it more closely, I don't really understand this fix. 
uic_mask_ack_irq() is just uic_mask_irq() followed by uic_ack_irq(). 
So, why do you remove the ipipe_irq_lock() from uic_mask_ack_irq() and
not from uic_mask_irq()?  Also, since uic_unmask_irq() always calls
ipipe_irq_unlock(), doesn't that mean you can wind up with more unlocks
than locks (since uic_mask_ack_irq/uic_unmask_irq are no longer balanced)?

>> Also, please include my compile-time patch, if that is acceptible.
>>
>>     
>
> I recently committed a different fix for the same issue after you reported it.
> This should work without requiring additional #ifdef'ing as well. Thanks for the
> heads up.
>
> --- a/include/asm-powerpc/ipipe.h
> +++ b/include/asm-powerpc/ipipe.h
> @@ -62,8 +62,6 @@
>  		local_irq_enable_hw(); x;				\
>  	} )
>
> -#define ipipe_update_tick_evtdev(evtdev)	do { } while (0)
> -
>  struct ipipe_domain;
>
>  struct ipipe_sysinfo {
> @@ -209,4 +207,6 @@ do {									\
>
>  #endif /* CONFIG_IPIPE */
>
> +#define ipipe_update_tick_evtdev(evtdev)	do { } while (0)
> +
>  #endif /* !__ASM_POWERPC_IPIPE_H */
>   

Ok - thanks.  I'll apply that fix instead.

    Steve



[-- Attachment #2: Type: text/html, Size: 2442 bytes --]

Re: [Xenomai-core] Compile-time bug, and problem with PPC440 ethernet

[Philippe Gerum]

Steven A. Falco wrote:
> 
>>> Beautiful.  The patch works!  I now get DHCP replies.
>>>
>>> Is this the patch you will put into the official tree, or do you still
>>> need to do more?
>>>
>>>     
>>
>> One thing, could you confirm that your network card relies on edge interrupts
>> (and not level)?
>>
>> Aside of that, it should be ok. It's the same IRQ lock out issue fixed in the
>> ppc/ branch recently.
>>
>>   
> 
> Sorry for the delay in getting back to you - vacation. 
> 
> The ethernet interrupts on the PPC440 are level-sensitive.  I gather
> that is not what you were expecting, so I wonder what the implications are.
>

Nothing bad. This just means that the bug was due to unbalanced locking, which
is only ok when trying to decrease the lock count uselessly.

> After looking at it more closely, I don't really understand this fix. 
> uic_mask_ack_irq() is just uic_mask_irq() followed by uic_ack_irq(). 
> So, why do you remove the ipipe_irq_lock() from uic_mask_ack_irq() and
> not from uic_mask_irq()?

We don't want to lock the pipeline for a given interrupt when acknowledging that
interrupt -- when present uic_mask_ack_irq() will always supersede separate
mask+ack calls for acknowledging IRQs.

However, we want non ack-related masking/unmasking to alter the pipeline state
for the interrupt in question. And those routines are not involved in IRQ
acknowledge at all.

  Also, since uic_unmask_irq() always calls
> ipipe_irq_unlock(), doesn't that mean you can wind up with more unlocks
> than locks (since uic_mask_ack_irq/uic_unmask_irq are no longer balanced)?
>

That is explicitly allowed by the pipeline code.

>>> Also, please include my compile-time patch, if that is acceptible.
>>>
>>>     
>>
>> I recently committed a different fix for the same issue after you reported it.
>> This should work without requiring additional #ifdef'ing as well. Thanks for the
>> heads up.
>>
>> --- a/include/asm-powerpc/ipipe.h
>> +++ b/include/asm-powerpc/ipipe.h
>> @@ -62,8 +62,6 @@
>>  		local_irq_enable_hw(); x;				\
>>  	} )
>>
>> -#define ipipe_update_tick_evtdev(evtdev)	do { } while (0)
>> -
>>  struct ipipe_domain;
>>
>>  struct ipipe_sysinfo {
>> @@ -209,4 +207,6 @@ do {									\
>>
>>  #endif /* CONFIG_IPIPE */
>>
>> +#define ipipe_update_tick_evtdev(evtdev)	do { } while (0)
>> +
>>  #endif /* !__ASM_POWERPC_IPIPE_H */
>>   
> 
> Ok - thanks.  I'll apply that fix instead.
> 
>     Steve
> 
> 


-- 
Philippe.