[Xenomai] 3.14 hangs with ipipe patch applied

[Xenomai] 3.14 hangs with ipipe patch applied

[Lennart Sorensen]

I am trying to get 3.14.39 to boot with ipipe/xenomai.  The working
kernel boots fine, but once I apply the ipipe patch even without enabling
CONFIG_IPIPE, it stops at:

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 3.14-2-am5726 (debian-kernel@lists.debian.org) (gcc version 4.7.2 (Debian 4.7.2-5) ) #1 SMP Debian 3.14.39-0.1 (2015-04-28)
[    0.000000] CPU: ARMv7 Processor [412fc0f2] revision 2 (ARMv7), cr=10c5387d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache
[    0.000000] Machine model: RCM RX1400
[    0.000000] bootconsole [earlycon0] enabled
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] DRA752 ES1.1
[    0.000000] PERCPU: Embedded 8 pages/cpu @dfac0000 s11584 r8192 d12992 u32768
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 129792
[    0.000000] Kernel command line: root=/dev/mmcblk0p7 ro console=ttyS2,57600n8 rootwait mem=512M fips=1 single earlyprintk bootver=2014.04RR16
[    0.000000] fips mode: enabled
[    0.000000] PID hash table entries: 2048 (order: 1, 8192 bytes)
[    0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
[    0.000000] Memory: 510548K/523264K available (4604K kernel code, 531K rwdata, 1956K rodata, 451K init, 326K bss, 12716K reserved, 0K highmem)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
[    0.000000]     fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)
[    0.000000]     vmalloc : 0xe0800000 - 0xff000000   ( 488 MB)
[    0.000000]     lowmem  : 0xc0000000 - 0xe0000000   ( 512 MB)
[    0.000000]     pkmap   : 0xbfe00000 - 0xc0000000   (   2 MB)
[    0.000000]     modules : 0xbf000000 - 0xbfe00000   (  14 MB)
[    0.000000]       .text : 0xc0008000 - 0xc06703b4   (6561 kB)
[    0.000000]       .init : 0xc0671000 - 0xc06e1d40   ( 452 kB)
[    0.000000]       .data : 0xc06e2000 - 0xc0766c48   ( 532 kB)
[    0.000000]        .bss : 0xc0766c48 - 0xc07b87a0   ( 327 kB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000] NR_IRQS:16 nr_irqs:16 16
[    0.000000] kmemleak: Kernel memory leak detector disabled
[    0.000000] ti_dt_clocks_register: failed to lookup clock node pruss1
[    0.000000] ti_dt_clocks_register: failed to lookup clock node pruss2
[    0.000000] OMAP clockevent source: timer1 at 32786 Hz
[    0.000000] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 65536000000000ns
[    0.017242] OMAP clocksource: 32k_counter at 32768 Hz
[    0.028015] Architected cp15 timer(s) running at 6.14MHz (phys).
[    0.040470] sched_clock: 56 bits at 6MHz, resolution 162ns, wraps every 2794592043008ns
[    0.057000] Switching to timer-based delay loop

Remove the ipipe patch, and it boots fine.

I was doing fine with 3.12, so I wonder what changed in the 3.14 tree
that could be causing this.  This being the ti-linux-3.14.y tree due to
the CPU in use.

Any suggestion for what part of the code to look at to explain what
could be happening right after the timer is enabled?

On the none ipipe patched kernel the next line is:

[    0.167509] Calibrating delay loop (skipped), value calculated using timer frequency.. 12.29 BogoMIPS (lpj=6147)

so seemingly I am not hitting that for some reason.

-- 
Len Sorensen

Re: [Xenomai] 3.14 hangs with ipipe patch applied

[Gilles Chanteperdrix]

On Wed, May 13, 2015 at 04:33:01PM -0400, Lennart Sorensen wrote:
> Any suggestion for what part of the code to look at to explain what
> could be happening right after the timer is enabled?

Sorry, nothing more to suggest than what is already in the porting guide.

-- 
					    Gilles.

Re: [Xenomai] 3.14 hangs with ipipe patch applied

[Lennart Sorensen]

On Wed, May 13, 2015 at 10:37:43PM +0200, Gilles Chanteperdrix wrote:
> On Wed, May 13, 2015 at 04:33:01PM -0400, Lennart Sorensen wrote:
> > Any suggestion for what part of the code to look at to explain what
> > could be happening right after the timer is enabled?
> 
> Sorry, nothing more to suggest than what is already in the porting guide.

I am trying to check that the 3.12 and 3.14 configs don't have some
stupid difference I missed.

-- 
Len Sorensen

Re: [Xenomai] 3.14 hangs with ipipe patch applied

[Lennart Sorensen]

On Wed, May 13, 2015 at 04:38:33PM -0400, Lennart Sorensen wrote:
> On Wed, May 13, 2015 at 10:37:43PM +0200, Gilles Chanteperdrix wrote:
> > On Wed, May 13, 2015 at 04:33:01PM -0400, Lennart Sorensen wrote:
> > > Any suggestion for what part of the code to look at to explain what
> > > could be happening right after the timer is enabled?
> > 
> > Sorry, nothing more to suggest than what is already in the porting guide.
> 
> I am trying to check that the 3.12 and 3.14 configs don't have some
> stupid difference I missed.

OK it is getting stuck here:

static bool hvc_dcc_check(void)
{
        unsigned long time = jiffies + (HZ / 10);

        /* Write a test character to check if it is handled */
        __dcc_putchar('\n');

// it gets here
        while (time_is_after_jiffies(time)) {
                if (!(__dcc_getstatus() & DCC_STATUS_TX))
                        return true;
        }
// it never gets here

        return false;
}

Any idea how the ipipe patch (even with CONFIG_IPIPE=n) causes that look
to get stuck forever?

-- 
Len Sorensen

Re: [Xenomai] 3.14 hangs with ipipe patch applied

[Gilles Chanteperdrix]

On Thu, May 14, 2015 at 12:23:51PM -0400, Lennart Sorensen wrote:
> On Wed, May 13, 2015 at 04:38:33PM -0400, Lennart Sorensen wrote:
> > On Wed, May 13, 2015 at 10:37:43PM +0200, Gilles Chanteperdrix wrote:
> > > On Wed, May 13, 2015 at 04:33:01PM -0400, Lennart Sorensen wrote:
> > > > Any suggestion for what part of the code to look at to explain what
> > > > could be happening right after the timer is enabled?
> > > 
> > > Sorry, nothing more to suggest than what is already in the porting guide.
> > 
> > I am trying to check that the 3.12 and 3.14 configs don't have some
> > stupid difference I missed.
> 
> OK it is getting stuck here:
> 
> static bool hvc_dcc_check(void)
> {
>         unsigned long time = jiffies + (HZ / 10);
> 
>         /* Write a test character to check if it is handled */
>         __dcc_putchar('\n');
> 
> // it gets here
>         while (time_is_after_jiffies(time)) {
>                 if (!(__dcc_getstatus() & DCC_STATUS_TX))
>                         return true;
>         }
> // it never gets here
> 
>         return false;
> }
> 
> Any idea how the ipipe patch (even with CONFIG_IPIPE=n) causes that look
> to get stuck forever?

The DCC should not be used with SMP for the obvious reason shown by
this code. I observed bugs on Zynq without the I-pipe patch.

Now the second obvious thing is that if time_is_after_jiffies() does
not make progress, it is because the timer does not tick. Once
again, how to debug this is explained in the troubleshooting section
of the porting guide.

-- 
					    Gilles.

Re: [Xenomai] 3.14 hangs with ipipe patch applied

[Lennart Sorensen]

On Thu, May 14, 2015 at 06:28:31PM +0200, Gilles Chanteperdrix wrote:
> On Thu, May 14, 2015 at 12:23:51PM -0400, Lennart Sorensen wrote:
> > On Wed, May 13, 2015 at 04:38:33PM -0400, Lennart Sorensen wrote:
> > > On Wed, May 13, 2015 at 10:37:43PM +0200, Gilles Chanteperdrix wrote:
> > > > On Wed, May 13, 2015 at 04:33:01PM -0400, Lennart Sorensen wrote:
> > > > > Any suggestion for what part of the code to look at to explain what
> > > > > could be happening right after the timer is enabled?
> > > > 
> > > > Sorry, nothing more to suggest than what is already in the porting guide.
> > > 
> > > I am trying to check that the 3.12 and 3.14 configs don't have some
> > > stupid difference I missed.
> > 
> > OK it is getting stuck here:
> > 
> > static bool hvc_dcc_check(void)
> > {
> >         unsigned long time = jiffies + (HZ / 10);
> > 
> >         /* Write a test character to check if it is handled */
> >         __dcc_putchar('\n');
> > 
> > // it gets here
> >         while (time_is_after_jiffies(time)) {
> >                 if (!(__dcc_getstatus() & DCC_STATUS_TX))
> >                         return true;
> >         }
> > // it never gets here
> > 
> >         return false;
> > }
> > 
> > Any idea how the ipipe patch (even with CONFIG_IPIPE=n) causes that look
> > to get stuck forever?
> 
> The DCC should not be used with SMP for the obvious reason shown by
> this code. I observed bugs on Zynq without the I-pipe patch.
> 
> Now the second obvious thing is that if time_is_after_jiffies() does
> not make progress, it is because the timer does not tick. Once
> again, how to debug this is explained in the troubleshooting section
> of the porting guide.

OK, that makes sense.

Turning off the DCC (which I really have no need for) makes it get a
bit further before gettings stuck, so I think the timer being stuck is
indeed the correct theory to investigate.  Odd how 3.12 this worked fine.

[    2.383692] VFS: Disk quotas dquot_6.5.2
[    2.392599] Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[    2.411815] msgmni has been set to 1486
[    2.421043] alg: self-tests for aes-generic (aes) passed
[    2.432651] alg: self-tests for crc32c-generic (crc32c) passed
[    2.445092] alg: self-tests for crct10dif-generic (crct10dif) passed
[    2.458414] alg: No test for stdrng (krng)
[    2.467837] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)
[    2.483130] io scheduler noop registered
[    2.491314] io scheduler deadline registered (default)
[    2.502613] io scheduler cfq registered
[    2.516676] pinctrl-single 4a003400.pinmux: 281 pins at pa fc003400 size 1124
[    2.535869] pbias_mmc_omap5: 1800 <--> 3000 mV at 3000 mV 
[    2.770821] Serial: 8250/16550 driver, 16 ports, IRQ sharing disabled
[    2.797902] 4806a000.serial: ttyS0 at MMIO 0x4806a000 (irq = 296, base_baud = 3000000) is a 8250
[    2.817855] 48020000.serial: ttyS2 at MMIO 0x48020000 (irq = 297, base_baud = 3000000) is a 8250
[    2.836028] console [ttyS2] enabled
[    2.836028] console [ttyS2] enabled
[    2.850309] bootconsole [earlycon0] disabled
[    2.850309] bootconsole [earlycon0] disabled
[    2.870425] 48420000.serial: ttyS6 at MMIO 0x48420000 (irq = 298, base_baud = 3000000) is a 8250
[    2.889154] omap8250 48422000.serial: mode_gpio_request return -16
[    2.901601] omap8250: probe of 48422000.serial failed with error -16
[    2.914812] pru_rs485_init
[    2.921743] omap_rng 48090000.rng: OMAP Random Number Generator ver. 20
[    2.940130] libphy: Fixed MDIO Bus: probed
[    2.950381] usbcore: registered new interface driver cdc_wdm
[    2.962063] usbcore: registered new interface driver usb-storage
[    2.975517] omap_rtc 48838000.rtcss: rtc core: registered 48838000.rtcss as rtc0
[    2.991612] i2c /dev entries driver
[    3.000692] device-mapper: uevent: version 1.0.3
[    3.011097] device-mapper: ioctl: 4.27.0-ioctl (2013-10-30) initialised: dm-devel@redhat.com

and there it gets stuck again.

I will try to figure out what is wrong with the timer tick when the
ipipe patch is applied.

-- 
Len Sorensen

Re: [Xenomai] 3.14 hangs with ipipe patch applied

[Lennart Sorensen]

On Thu, May 14, 2015 at 12:47:45PM -0400, Lennart Sorensen wrote:
> OK, that makes sense.
> 
> Turning off the DCC (which I really have no need for) makes it get a
> bit further before gettings stuck, so I think the timer being stuck is
> indeed the correct theory to investigate.  Odd how 3.12 this worked fine.
> 
> [    2.383692] VFS: Disk quotas dquot_6.5.2
> [    2.392599] Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
> [    2.411815] msgmni has been set to 1486
> [    2.421043] alg: self-tests for aes-generic (aes) passed
> [    2.432651] alg: self-tests for crc32c-generic (crc32c) passed
> [    2.445092] alg: self-tests for crct10dif-generic (crct10dif) passed
> [    2.458414] alg: No test for stdrng (krng)
> [    2.467837] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)
> [    2.483130] io scheduler noop registered
> [    2.491314] io scheduler deadline registered (default)
> [    2.502613] io scheduler cfq registered
> [    2.516676] pinctrl-single 4a003400.pinmux: 281 pins at pa fc003400 size 1124
> [    2.535869] pbias_mmc_omap5: 1800 <--> 3000 mV at 3000 mV 
> [    2.770821] Serial: 8250/16550 driver, 16 ports, IRQ sharing disabled
> [    2.797902] 4806a000.serial: ttyS0 at MMIO 0x4806a000 (irq = 296, base_baud = 3000000) is a 8250
> [    2.817855] 48020000.serial: ttyS2 at MMIO 0x48020000 (irq = 297, base_baud = 3000000) is a 8250
> [    2.836028] console [ttyS2] enabled
> [    2.836028] console [ttyS2] enabled
> [    2.850309] bootconsole [earlycon0] disabled
> [    2.850309] bootconsole [earlycon0] disabled
> [    2.870425] 48420000.serial: ttyS6 at MMIO 0x48420000 (irq = 298, base_baud = 3000000) is a 8250
> [    2.889154] omap8250 48422000.serial: mode_gpio_request return -16
> [    2.901601] omap8250: probe of 48422000.serial failed with error -16
> [    2.914812] pru_rs485_init
> [    2.921743] omap_rng 48090000.rng: OMAP Random Number Generator ver. 20
> [    2.940130] libphy: Fixed MDIO Bus: probed
> [    2.950381] usbcore: registered new interface driver cdc_wdm
> [    2.962063] usbcore: registered new interface driver usb-storage
> [    2.975517] omap_rtc 48838000.rtcss: rtc core: registered 48838000.rtcss as rtc0
> [    2.991612] i2c /dev entries driver
> [    3.000692] device-mapper: uevent: version 1.0.3
> [    3.011097] device-mapper: ioctl: 4.27.0-ioctl (2013-10-30) initialised: dm-devel@redhat.com
> 
> and there it gets stuck again.
> 
> I will try to figure out what is wrong with the timer tick when the
> ipipe patch is applied.

I managed to confirm that jiffies is in fact stuck at the boot time
-300000 value.  Now if I could only track down what code is supposed to
increment it.  The code I found (do_timer called by xtime_update) does
not appear to get called by anyone.

-- 
Len Sorensen

Re: [Xenomai] 3.14 hangs with ipipe patch applied

[Lennart Sorensen]

On Thu, May 14, 2015 at 03:34:33PM -0400, Lennart Sorensen wrote:
> I managed to confirm that jiffies is in fact stuck at the boot time
> -300000 value.  Now if I could only track down what code is supposed to
> increment it.  The code I found (do_timer called by xtime_update) does
> not appear to get called by anyone.

I have narrowed it down.  If I apply the ipipe patch except for
drivers/irqchip/irq-gic.c then it still boots.

So only about 200 lines of patch left to investigate.

I guess the reason the timer ticks were not happening was that no
interrupts were happening.

-- 
Len Sorensen

Re: [Xenomai] 3.14 hangs with ipipe patch applied

[Gilles Chanteperdrix]

On Thu, May 21, 2015 at 12:32:08PM -0400, Lennart Sorensen wrote:
> On Thu, May 14, 2015 at 03:34:33PM -0400, Lennart Sorensen wrote:
> > I managed to confirm that jiffies is in fact stuck at the boot time
> > -300000 value.  Now if I could only track down what code is supposed to
> > increment it.  The code I found (do_timer called by xtime_update) does
> > not appear to get called by anyone.
> 
> I have narrowed it down.  If I apply the ipipe patch except for
> drivers/irqchip/irq-gic.c then it still boots.
> 
> So only about 200 lines of patch left to investigate.
> 
> I guess the reason the timer ticks were not happening was that no
> interrupts were happening.

There is no need to guess that, it can be verified easily.

-- 
					    Gilles.

Re: [Xenomai] 3.14 hangs with ipipe patch applied

[Lennart Sorensen]

On Thu, May 21, 2015 at 06:34:40PM +0200, Gilles Chanteperdrix wrote:
> On Thu, May 21, 2015 at 12:32:08PM -0400, Lennart Sorensen wrote:
> > On Thu, May 14, 2015 at 03:34:33PM -0400, Lennart Sorensen wrote:
> > > I managed to confirm that jiffies is in fact stuck at the boot time
> > > -300000 value.  Now if I could only track down what code is supposed to
> > > increment it.  The code I found (do_timer called by xtime_update) does
> > > not appear to get called by anyone.
> > 
> > I have narrowed it down.  If I apply the ipipe patch except for
> > drivers/irqchip/irq-gic.c then it still boots.
> > 
> > So only about 200 lines of patch left to investigate.
> > 
> > I guess the reason the timer ticks were not happening was that no
> > interrupts were happening.
> 
> There is no need to guess that, it can be verified easily.

This seems to solve my problem:

root@lennartsorensen-386:~/linux-3.14-wip/linux-3.14.39.rr1# diff -u debian/patches/ipipe-core-3.14.39.diff.original debian/patches/ipipe-core-3.14.39.diff
--- debian/patches/ipipe-core-3.14.39.diff.original     2015-05-21 12:32:53.669255161 -0400
+++ debian/patches/ipipe-core-3.14.39.diff      2015-05-21 12:50:20.096497809 -0400
@@ -24605,12 +24605,12 @@
  static void gic_mask_irq(struct irq_data *d)
  {
 -      u32 mask = 1 << (gic_irq(d) % 32);
-+      u32 mask = 1 << (d->irq % 32);
++      u32 mask = 1 << (gic_irq(d) % 32);
 +      unsigned long flags;
  
 -      raw_spin_lock(&irq_controller_lock);
 +      raw_spin_lock_irqsave_cond(&irq_controller_lock, flags);
-+      ipipe_lock_irq(d->irq);
++      ipipe_lock_irq(gic_irq(d));
        writel_relaxed(mask, gic_dist_base(d) + GIC_DIST_ENABLE_CLEAR + (gic_irq(d) / 32) * 4);
        if (gic_arch_extn.irq_mask)
                gic_arch_extn.irq_mask(d);
@@ -24621,7 +24621,7 @@
  static void gic_unmask_irq(struct irq_data *d)
  {
 -      u32 mask = 1 << (gic_irq(d) % 32);
-+      u32 mask = 1 << (d->irq % 32);
++      u32 mask = 1 << (gic_irq(d) % 32);
 +      unsigned long flags;
  
 -      raw_spin_lock(&irq_controller_lock);
@@ -24630,7 +24630,7 @@
                gic_arch_extn.irq_unmask(d);
        writel_relaxed(mask, gic_dist_base(d) + GIC_DIST_ENABLE_SET + (gic_irq(d) / 32) * 4);
 -      raw_spin_unlock(&irq_controller_lock);
-+      ipipe_unlock_irq(d->irq);
++      ipipe_unlock_irq(gic_irq(d));
 +      raw_spin_unlock_irqrestore_cond(&irq_controller_lock, flags);
  }
  
@@ -24652,7 +24652,7 @@
 +
 +static void gic_hold_irq(struct irq_data *d)
 +{
-+      u32 mask = 1 << (d->irq % 32);
++      u32 mask = 1 << (gic_irq(d) % 32);
 +      unsigned long flags;
 +
 +      raw_spin_lock_irqsave_cond(&irq_controller_lock, flags);
@@ -24668,7 +24668,7 @@
 +
 +static void gic_release_irq(struct irq_data *d)
 +{
-+      u32 mask = 1 << (d->irq % 32);
++      u32 mask = 1 << (gic_irq(d) % 32);
 +      unsigned long flags;
 +
 +      raw_spin_lock_irqsave_cond(&irq_controller_lock, flags);

gic_irq(d) returns d->hwirq, which is not the same as d->irq (or at
least does not have to be, but probably is on many systems).

Does this make sense?

-- 
Len Sorensen

Re: [Xenomai] 3.14 hangs with ipipe patch applied

[Gilles Chanteperdrix]

On Thu, May 21, 2015 at 12:59:13PM -0400, Lennart Sorensen wrote:
> On Thu, May 21, 2015 at 06:34:40PM +0200, Gilles Chanteperdrix wrote:
> > On Thu, May 21, 2015 at 12:32:08PM -0400, Lennart Sorensen wrote:
> > > On Thu, May 14, 2015 at 03:34:33PM -0400, Lennart Sorensen wrote:
> > > > I managed to confirm that jiffies is in fact stuck at the boot time
> > > > -300000 value.  Now if I could only track down what code is supposed to
> > > > increment it.  The code I found (do_timer called by xtime_update) does
> > > > not appear to get called by anyone.
> > > 
> > > I have narrowed it down.  If I apply the ipipe patch except for
> > > drivers/irqchip/irq-gic.c then it still boots.
> > > 
> > > So only about 200 lines of patch left to investigate.
> > > 
> > > I guess the reason the timer ticks were not happening was that no
> > > interrupts were happening.
> > 
> > There is no need to guess that, it can be verified easily.
> 
> This seems to solve my problem:
> 
> root@lennartsorensen-386:~/linux-3.14-wip/linux-3.14.39.rr1# diff -u debian/patches/ipipe-core-3.14.39.diff.original debian/patches/ipipe-core-3.14.39.diff
> --- debian/patches/ipipe-core-3.14.39.diff.original     2015-05-21 12:32:53.669255161 -0400
> +++ debian/patches/ipipe-core-3.14.39.diff      2015-05-21 12:50:20.096497809 -0400
> @@ -24605,12 +24605,12 @@
>   static void gic_mask_irq(struct irq_data *d)
>   {
>  -      u32 mask = 1 << (gic_irq(d) % 32);
> -+      u32 mask = 1 << (d->irq % 32);
> ++      u32 mask = 1 << (gic_irq(d) % 32);
>  +      unsigned long flags;
>   
>  -      raw_spin_lock(&irq_controller_lock);
>  +      raw_spin_lock_irqsave_cond(&irq_controller_lock, flags);
> -+      ipipe_lock_irq(d->irq);
> ++      ipipe_lock_irq(gic_irq(d));
>         writel_relaxed(mask, gic_dist_base(d) + GIC_DIST_ENABLE_CLEAR + (gic_irq(d) / 32) * 4);
>         if (gic_arch_extn.irq_mask)
>                 gic_arch_extn.irq_mask(d);
> @@ -24621,7 +24621,7 @@
>   static void gic_unmask_irq(struct irq_data *d)
>   {
>  -      u32 mask = 1 << (gic_irq(d) % 32);
> -+      u32 mask = 1 << (d->irq % 32);
> ++      u32 mask = 1 << (gic_irq(d) % 32);
>  +      unsigned long flags;
>   
>  -      raw_spin_lock(&irq_controller_lock);
> @@ -24630,7 +24630,7 @@
>                 gic_arch_extn.irq_unmask(d);
>         writel_relaxed(mask, gic_dist_base(d) + GIC_DIST_ENABLE_SET + (gic_irq(d) / 32) * 4);
>  -      raw_spin_unlock(&irq_controller_lock);
> -+      ipipe_unlock_irq(d->irq);
> ++      ipipe_unlock_irq(gic_irq(d));
>  +      raw_spin_unlock_irqrestore_cond(&irq_controller_lock, flags);
>   }
>   
> @@ -24652,7 +24652,7 @@
>  +
>  +static void gic_hold_irq(struct irq_data *d)
>  +{
> -+      u32 mask = 1 << (d->irq % 32);
> ++      u32 mask = 1 << (gic_irq(d) % 32);
>  +      unsigned long flags;
>  +
>  +      raw_spin_lock_irqsave_cond(&irq_controller_lock, flags);
> @@ -24668,7 +24668,7 @@
>  +
>  +static void gic_release_irq(struct irq_data *d)
>  +{
> -+      u32 mask = 1 << (d->irq % 32);
> ++      u32 mask = 1 << (gic_irq(d) % 32);
>  +      unsigned long flags;
>  +
>  +      raw_spin_lock_irqsave_cond(&irq_controller_lock, flags);
> 
> gic_irq(d) returns d->hwirq, which is not the same as d->irq (or at
> least does not have to be, but probably is on many systems).
> 
> Does this make sense?

Half of it. What should be passed to writel_relaxed and so on
should be based on the hw irq. But the argument of ipipe_lock_irq
for instance, should not be an hw irq.

-- 
					    Gilles.

Re: [Xenomai] 3.14 hangs with ipipe patch applied

[Lennart Sorensen]

On Thu, May 21, 2015 at 07:03:01PM +0200, Gilles Chanteperdrix wrote:
> On Thu, May 21, 2015 at 12:59:13PM -0400, Lennart Sorensen wrote:
> > On Thu, May 21, 2015 at 06:34:40PM +0200, Gilles Chanteperdrix wrote:
> > > On Thu, May 21, 2015 at 12:32:08PM -0400, Lennart Sorensen wrote:
> > > > On Thu, May 14, 2015 at 03:34:33PM -0400, Lennart Sorensen wrote:
> > > > > I managed to confirm that jiffies is in fact stuck at the boot time
> > > > > -300000 value.  Now if I could only track down what code is supposed to
> > > > > increment it.  The code I found (do_timer called by xtime_update) does
> > > > > not appear to get called by anyone.
> > > > 
> > > > I have narrowed it down.  If I apply the ipipe patch except for
> > > > drivers/irqchip/irq-gic.c then it still boots.
> > > > 
> > > > So only about 200 lines of patch left to investigate.
> > > > 
> > > > I guess the reason the timer ticks were not happening was that no
> > > > interrupts were happening.
> > > 
> > > There is no need to guess that, it can be verified easily.
> > 
> > This seems to solve my problem:
> > 
> > root@lennartsorensen-386:~/linux-3.14-wip/linux-3.14.39.rr1# diff -u debian/patches/ipipe-core-3.14.39.diff.original debian/patches/ipipe-core-3.14.39.diff
> > --- debian/patches/ipipe-core-3.14.39.diff.original     2015-05-21 12:32:53.669255161 -0400
> > +++ debian/patches/ipipe-core-3.14.39.diff      2015-05-21 12:50:20.096497809 -0400
> > @@ -24605,12 +24605,12 @@
> >   static void gic_mask_irq(struct irq_data *d)
> >   {
> >  -      u32 mask = 1 << (gic_irq(d) % 32);
> > -+      u32 mask = 1 << (d->irq % 32);
> > ++      u32 mask = 1 << (gic_irq(d) % 32);
> >  +      unsigned long flags;
> >   
> >  -      raw_spin_lock(&irq_controller_lock);
> >  +      raw_spin_lock_irqsave_cond(&irq_controller_lock, flags);
> > -+      ipipe_lock_irq(d->irq);
> > ++      ipipe_lock_irq(gic_irq(d));
> >         writel_relaxed(mask, gic_dist_base(d) + GIC_DIST_ENABLE_CLEAR + (gic_irq(d) / 32) * 4);
> >         if (gic_arch_extn.irq_mask)
> >                 gic_arch_extn.irq_mask(d);
> > @@ -24621,7 +24621,7 @@
> >   static void gic_unmask_irq(struct irq_data *d)
> >   {
> >  -      u32 mask = 1 << (gic_irq(d) % 32);
> > -+      u32 mask = 1 << (d->irq % 32);
> > ++      u32 mask = 1 << (gic_irq(d) % 32);
> >  +      unsigned long flags;
> >   
> >  -      raw_spin_lock(&irq_controller_lock);
> > @@ -24630,7 +24630,7 @@
> >                 gic_arch_extn.irq_unmask(d);
> >         writel_relaxed(mask, gic_dist_base(d) + GIC_DIST_ENABLE_SET + (gic_irq(d) / 32) * 4);
> >  -      raw_spin_unlock(&irq_controller_lock);
> > -+      ipipe_unlock_irq(d->irq);
> > ++      ipipe_unlock_irq(gic_irq(d));
> >  +      raw_spin_unlock_irqrestore_cond(&irq_controller_lock, flags);
> >   }
> >   
> > @@ -24652,7 +24652,7 @@
> >  +
> >  +static void gic_hold_irq(struct irq_data *d)
> >  +{
> > -+      u32 mask = 1 << (d->irq % 32);
> > ++      u32 mask = 1 << (gic_irq(d) % 32);
> >  +      unsigned long flags;
> >  +
> >  +      raw_spin_lock_irqsave_cond(&irq_controller_lock, flags);
> > @@ -24668,7 +24668,7 @@
> >  +
> >  +static void gic_release_irq(struct irq_data *d)
> >  +{
> > -+      u32 mask = 1 << (d->irq % 32);
> > ++      u32 mask = 1 << (gic_irq(d) % 32);
> >  +      unsigned long flags;
> >  +
> >  +      raw_spin_lock_irqsave_cond(&irq_controller_lock, flags);
> > 
> > gic_irq(d) returns d->hwirq, which is not the same as d->irq (or at
> > least does not have to be, but probably is on many systems).
> > 
> > Does this make sense?
> 
> Half of it. What should be passed to writel_relaxed and so on
> should be based on the hw irq. But the argument of ipipe_lock_irq
> for instance, should not be an hw irq.

OK, I was not sure, hence why I wanted to ask.

Let me fix that up and try that.  Then I should do a build with
CONFIG_IPIPE=y and see if that then starts to work.

At least this is progress.

-- 
Len Sorensen

Re: [Xenomai] 3.14 hangs with ipipe patch applied

[Lennart Sorensen]

On Thu, May 21, 2015 at 01:04:32PM -0400, Lennart Sorensen wrote:
> OK, I was not sure, hence why I wanted to ask.
> 
> Let me fix that up and try that.  Then I should do a build with
> CONFIG_IPIPE=y and see if that then starts to work.
> 
> At least this is progress.

So this perhaps:

--- debian/patches/ipipe-core-3.14.39.diff.original     2015-05-21 12:32:53.669255161 -0400
+++ debian/patches/ipipe-core-3.14.39.diff      2015-05-21 13:06:12.075112664 -0400
@@ -24605,7 +24605,7 @@
  static void gic_mask_irq(struct irq_data *d)
  {
 -      u32 mask = 1 << (gic_irq(d) % 32);
-+      u32 mask = 1 << (d->irq % 32);
++      u32 mask = 1 << (gic_irq(d) % 32);
 +      unsigned long flags;
  
 -      raw_spin_lock(&irq_controller_lock);
@@ -24621,7 +24621,7 @@
  static void gic_unmask_irq(struct irq_data *d)
  {
 -      u32 mask = 1 << (gic_irq(d) % 32);
-+      u32 mask = 1 << (d->irq % 32);
++      u32 mask = 1 << (gic_irq(d) % 32);
 +      unsigned long flags;
  
 -      raw_spin_lock(&irq_controller_lock);
@@ -24652,7 +24652,7 @@
 +
 +static void gic_hold_irq(struct irq_data *d)
 +{
-+      u32 mask = 1 << (d->irq % 32);
++      u32 mask = 1 << (gic_irq(d) % 32);
 +      unsigned long flags;
 +
 +      raw_spin_lock_irqsave_cond(&irq_controller_lock, flags);
@@ -24668,7 +24668,7 @@
 +
 +static void gic_release_irq(struct irq_data *d)
 +{
-+      u32 mask = 1 << (d->irq % 32);
++      u32 mask = 1 << (gic_irq(d) % 32);
 +      unsigned long flags;
 +
 +      raw_spin_lock_irqsave_cond(&irq_controller_lock, flags);

-- 
Len Sorensen

Re: [Xenomai] 3.14 hangs with ipipe patch applied

[Gilles Chanteperdrix]

On Thu, May 21, 2015 at 01:07:16PM -0400, Lennart Sorensen wrote:
> On Thu, May 21, 2015 at 01:04:32PM -0400, Lennart Sorensen wrote:
> > OK, I was not sure, hence why I wanted to ask.
> > 
> > Let me fix that up and try that.  Then I should do a build with
> > CONFIG_IPIPE=y and see if that then starts to work.
> > 
> > At least this is progress.
> 
> So this perhaps:
> 
> --- debian/patches/ipipe-core-3.14.39.diff.original     2015-05-21 12:32:53.669255161 -0400
> +++ debian/patches/ipipe-core-3.14.39.diff      2015-05-21 13:06:12.075112664 -0400
> @@ -24605,7 +24605,7 @@
>   static void gic_mask_irq(struct irq_data *d)
>   {
>  -      u32 mask = 1 << (gic_irq(d) % 32);
> -+      u32 mask = 1 << (d->irq % 32);
> ++      u32 mask = 1 << (gic_irq(d) % 32);
>  +      unsigned long flags;
>   
>  -      raw_spin_lock(&irq_controller_lock);
> @@ -24621,7 +24621,7 @@
>   static void gic_unmask_irq(struct irq_data *d)
>   {
>  -      u32 mask = 1 << (gic_irq(d) % 32);
> -+      u32 mask = 1 << (d->irq % 32);
> ++      u32 mask = 1 << (gic_irq(d) % 32);
>  +      unsigned long flags;
>   
>  -      raw_spin_lock(&irq_controller_lock);
> @@ -24652,7 +24652,7 @@
>  +
>  +static void gic_hold_irq(struct irq_data *d)
>  +{
> -+      u32 mask = 1 << (d->irq % 32);
> ++      u32 mask = 1 << (gic_irq(d) % 32);
>  +      unsigned long flags;
>  +
>  +      raw_spin_lock_irqsave_cond(&irq_controller_lock, flags);
> @@ -24668,7 +24668,7 @@
>  +
>  +static void gic_release_irq(struct irq_data *d)
>  +{
> -+      u32 mask = 1 << (d->irq % 32);
> ++      u32 mask = 1 << (gic_irq(d) % 32);
>  +      unsigned long flags;
>  +
>  +      raw_spin_lock_irqsave_cond(&irq_controller_lock, flags);

Actually, gic_mask_irq and gic_unmask_irq should be identical to the
unpatched kernel with regard to registers reads and writes. And
gic_hold_irq/gic_release_irq should be identical to gic_mask/unmask.

-- 
					    Gilles.

Re: [Xenomai] 3.14 hangs with ipipe patch applied

[Lennart Sorensen]

On Thu, May 21, 2015 at 07:10:18PM +0200, Gilles Chanteperdrix wrote:
> On Thu, May 21, 2015 at 01:07:16PM -0400, Lennart Sorensen wrote:
> > On Thu, May 21, 2015 at 01:04:32PM -0400, Lennart Sorensen wrote:
> > > OK, I was not sure, hence why I wanted to ask.
> > > 
> > > Let me fix that up and try that.  Then I should do a build with
> > > CONFIG_IPIPE=y and see if that then starts to work.
> > > 
> > > At least this is progress.
> > 
> > So this perhaps:
> > 
> > --- debian/patches/ipipe-core-3.14.39.diff.original     2015-05-21 12:32:53.669255161 -0400
> > +++ debian/patches/ipipe-core-3.14.39.diff      2015-05-21 13:06:12.075112664 -0400
> > @@ -24605,7 +24605,7 @@
> >   static void gic_mask_irq(struct irq_data *d)
> >   {
> >  -      u32 mask = 1 << (gic_irq(d) % 32);
> > -+      u32 mask = 1 << (d->irq % 32);
> > ++      u32 mask = 1 << (gic_irq(d) % 32);
> >  +      unsigned long flags;
> >   
> >  -      raw_spin_lock(&irq_controller_lock);
> > @@ -24621,7 +24621,7 @@
> >   static void gic_unmask_irq(struct irq_data *d)
> >   {
> >  -      u32 mask = 1 << (gic_irq(d) % 32);
> > -+      u32 mask = 1 << (d->irq % 32);
> > ++      u32 mask = 1 << (gic_irq(d) % 32);
> >  +      unsigned long flags;
> >   
> >  -      raw_spin_lock(&irq_controller_lock);
> > @@ -24652,7 +24652,7 @@
> >  +
> >  +static void gic_hold_irq(struct irq_data *d)
> >  +{
> > -+      u32 mask = 1 << (d->irq % 32);
> > ++      u32 mask = 1 << (gic_irq(d) % 32);
> >  +      unsigned long flags;
> >  +
> >  +      raw_spin_lock_irqsave_cond(&irq_controller_lock, flags);
> > @@ -24668,7 +24668,7 @@
> >  +
> >  +static void gic_release_irq(struct irq_data *d)
> >  +{
> > -+      u32 mask = 1 << (d->irq % 32);
> > ++      u32 mask = 1 << (gic_irq(d) % 32);
> >  +      unsigned long flags;
> >  +
> >  +      raw_spin_lock_irqsave_cond(&irq_controller_lock, flags);
> 
> Actually, gic_mask_irq and gic_unmask_irq should be identical to the
> unpatched kernel with regard to registers reads and writes. And
> gic_hold_irq/gic_release_irq should be identical to gic_mask/unmask.

OK, I will go compare to those.

Thanks.

-- 
Len Sorensen

Re: [Xenomai] 3.14 hangs with ipipe patch applied

[Lennart Sorensen]

On Thu, May 21, 2015 at 07:10:18PM +0200, Gilles Chanteperdrix wrote:
> On Thu, May 21, 2015 at 01:07:16PM -0400, Lennart Sorensen wrote:
> > On Thu, May 21, 2015 at 01:04:32PM -0400, Lennart Sorensen wrote:
> > > OK, I was not sure, hence why I wanted to ask.
> > > 
> > > Let me fix that up and try that.  Then I should do a build with
> > > CONFIG_IPIPE=y and see if that then starts to work.
> > > 
> > > At least this is progress.
> > 
> > So this perhaps:
> > 
> > --- debian/patches/ipipe-core-3.14.39.diff.original     2015-05-21 12:32:53.669255161 -0400
> > +++ debian/patches/ipipe-core-3.14.39.diff      2015-05-21 13:06:12.075112664 -0400
> > @@ -24605,7 +24605,7 @@
> >   static void gic_mask_irq(struct irq_data *d)
> >   {
> >  -      u32 mask = 1 << (gic_irq(d) % 32);
> > -+      u32 mask = 1 << (d->irq % 32);
> > ++      u32 mask = 1 << (gic_irq(d) % 32);
> >  +      unsigned long flags;
> >   
> >  -      raw_spin_lock(&irq_controller_lock);
> > @@ -24621,7 +24621,7 @@
> >   static void gic_unmask_irq(struct irq_data *d)
> >   {
> >  -      u32 mask = 1 << (gic_irq(d) % 32);
> > -+      u32 mask = 1 << (d->irq % 32);
> > ++      u32 mask = 1 << (gic_irq(d) % 32);
> >  +      unsigned long flags;
> >   
> >  -      raw_spin_lock(&irq_controller_lock);
> > @@ -24652,7 +24652,7 @@
> >  +
> >  +static void gic_hold_irq(struct irq_data *d)
> >  +{
> > -+      u32 mask = 1 << (d->irq % 32);
> > ++      u32 mask = 1 << (gic_irq(d) % 32);
> >  +      unsigned long flags;
> >  +
> >  +      raw_spin_lock_irqsave_cond(&irq_controller_lock, flags);
> > @@ -24668,7 +24668,7 @@
> >  +
> >  +static void gic_release_irq(struct irq_data *d)
> >  +{
> > -+      u32 mask = 1 << (d->irq % 32);
> > ++      u32 mask = 1 << (gic_irq(d) % 32);
> >  +      unsigned long flags;
> >  +
> >  +      raw_spin_lock_irqsave_cond(&irq_controller_lock, flags);
> 
> Actually, gic_mask_irq and gic_unmask_irq should be identical to the
> unpatched kernel with regard to registers reads and writes. And
> gic_hold_irq/gic_release_irq should be identical to gic_mask/unmask.

Well I am reverting the change to gic_mask_irq and gic_unmask_irq, so
that should make them identical to the unpatched kernel, and that does
in fact make the system now boot.

And I am changing the mask in gic_hold_irq/gic_release_irq to match
gic_mask_irq/gic_unmask_irq, so that seems like it should be correct
too then.

Should I make a proper patch based on the current 3.14 git tree to make
it easier to read?  diffs of diffs are a bit messy.

-- 
Len Sorensen

Re: [Xenomai] 3.14 hangs with ipipe patch applied

[Gilles Chanteperdrix]

On Thu, May 21, 2015 at 01:16:41PM -0400, Lennart Sorensen wrote:
> On Thu, May 21, 2015 at 07:10:18PM +0200, Gilles Chanteperdrix wrote:
> > On Thu, May 21, 2015 at 01:07:16PM -0400, Lennart Sorensen wrote:
> > > On Thu, May 21, 2015 at 01:04:32PM -0400, Lennart Sorensen wrote:
> > > > OK, I was not sure, hence why I wanted to ask.
> > > > 
> > > > Let me fix that up and try that.  Then I should do a build with
> > > > CONFIG_IPIPE=y and see if that then starts to work.
> > > > 
> > > > At least this is progress.
> > > 
> > > So this perhaps:
> > > 
> > > --- debian/patches/ipipe-core-3.14.39.diff.original     2015-05-21 12:32:53.669255161 -0400
> > > +++ debian/patches/ipipe-core-3.14.39.diff      2015-05-21 13:06:12.075112664 -0400
> > > @@ -24605,7 +24605,7 @@
> > >   static void gic_mask_irq(struct irq_data *d)
> > >   {
> > >  -      u32 mask = 1 << (gic_irq(d) % 32);
> > > -+      u32 mask = 1 << (d->irq % 32);
> > > ++      u32 mask = 1 << (gic_irq(d) % 32);
> > >  +      unsigned long flags;
> > >   
> > >  -      raw_spin_lock(&irq_controller_lock);
> > > @@ -24621,7 +24621,7 @@
> > >   static void gic_unmask_irq(struct irq_data *d)
> > >   {
> > >  -      u32 mask = 1 << (gic_irq(d) % 32);
> > > -+      u32 mask = 1 << (d->irq % 32);
> > > ++      u32 mask = 1 << (gic_irq(d) % 32);
> > >  +      unsigned long flags;
> > >   
> > >  -      raw_spin_lock(&irq_controller_lock);
> > > @@ -24652,7 +24652,7 @@
> > >  +
> > >  +static void gic_hold_irq(struct irq_data *d)
> > >  +{
> > > -+      u32 mask = 1 << (d->irq % 32);
> > > ++      u32 mask = 1 << (gic_irq(d) % 32);
> > >  +      unsigned long flags;
> > >  +
> > >  +      raw_spin_lock_irqsave_cond(&irq_controller_lock, flags);
> > > @@ -24668,7 +24668,7 @@
> > >  +
> > >  +static void gic_release_irq(struct irq_data *d)
> > >  +{
> > > -+      u32 mask = 1 << (d->irq % 32);
> > > ++      u32 mask = 1 << (gic_irq(d) % 32);
> > >  +      unsigned long flags;
> > >  +
> > >  +      raw_spin_lock_irqsave_cond(&irq_controller_lock, flags);
> > 
> > Actually, gic_mask_irq and gic_unmask_irq should be identical to the
> > unpatched kernel with regard to registers reads and writes. And
> > gic_hold_irq/gic_release_irq should be identical to gic_mask/unmask.
> 
> Well I am reverting the change to gic_mask_irq and gic_unmask_irq, so
> that should make them identical to the unpatched kernel, and that does
> in fact make the system now boot.

Of course, you should keep the addition of ipipe_lock/unlock irq.
which is the point of patching these functions.

> 
> And I am changing the mask in gic_hold_irq/gic_release_irq to match
> gic_mask_irq/gic_unmask_irq, so that seems like it should be correct
> too then.
> 
> Should I make a proper patch based on the current 3.14 git tree to make
> it easier to read?  diffs of diffs are a bit messy.

If you are working with something more recent like 3.16 or 3.18,
please post as a patch to these.

-- 
					    Gilles.

Re: [Xenomai] 3.14 hangs with ipipe patch applied

[Lennart Sorensen]

On Thu, May 21, 2015 at 07:20:58PM +0200, Gilles Chanteperdrix wrote:
> Of course, you should keep the addition of ipipe_lock/unlock irq.
> which is the point of patching these functions.

Yes those I am not changing.

> If you are working with something more recent like 3.16 or 3.18,
> please post as a patch to these.

I am stuck at 3.14 since that's the current tree TI works with for
the am572x.

But I could generate a patch for the other ones, even if I can't test
it yet.

Should I send patches as a new thread?

-- 
Len Sorensen

Re: [Xenomai] 3.14 hangs with ipipe patch applied

[Lennart Sorensen]

On Thu, May 21, 2015 at 01:26:24PM -0400, Lennart Sorensen wrote:
> I am stuck at 3.14 since that's the current tree TI works with for
> the am572x.
> 
> But I could generate a patch for the other ones, even if I can't test
> it yet.
> 
> Should I send patches as a new thread?

So with CONFIG_IPIPE and CONFIG_XENOMAI enabled it still boots to
userspace, so yay!  I can continue getting things going now.

-- 
Len Sorensen