[Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Alexey Galakhov]

Hi,

I'm trying to run Xenomai on MINI2440 board. I use vanilla 2.6.35.9
kernel patched with adeos-ipipe-2.6.35.9-arm-1.18-01.patch . In kernel
configuration Xenomai is disabled, the only option (except debug) I
turned on is CONFIG_IPIPE. It hangs.

I suspect it is somehow related to TIMER3 which is used along with
TIMER4 on Samsung ARMs. Any suggestions?

My dmesg (with debugging on):

[    0.000000] Linux version 2.6.35.9-mini2440-rt
(agalakhov@domain.hid) (gcc version 4.6.0 (GCC) ) #9 PREEMPT Wed
Apr 27 18:33:31 YEKST 2011
[    0.000000] CPU: ARM920T [41129200] revision 0 (ARMv4T), cr=c0007177
[    0.000000] CPU: VIVT data cache, VIVT instruction cache
[    0.000000] Machine: MINI2440
[    0.000000] Memory policy: ECC disabled, Data cache writeback
[    0.000000] On node 0 totalpages: 16384
[    0.000000] free_area_init_node: node 0, pgdat c066ded4, node_mem_map
c06ee000
[    0.000000]   Normal zone: 128 pages used for memmap
[    0.000000]   Normal zone: 0 pages reserved
[    0.000000]   Normal zone: 16256 pages, LIFO batch:3
[    0.000000] CPU S3C2440A (id 0x32440001)
[    0.000000] S3C24XX Clocks, Copyright 2004 Simtec Electronics
[    0.000000] S3C244X: core 405.000 MHz, memory 101.250 MHz, peripheral
50.625 MHz
[    0.000000] CLOCK: Slow mode (1.500 MHz), fast, MPLL on, UPLL on
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on. 
Total pages: 16256
[    0.000000] Kernel command line: mini2440=1tb console=ttySAC0,115200
noinitrd rootwait rootfstype=ext2 root=/dev/mmcblk0p1 ro loglevel=8
[    0.000000] PID hash table entries: 256 (order: -2, 1024 bytes)
[    0.000000] Dentry cache hash table entries: 8192 (order: 3, 32768 bytes)
[    0.000000] Inode-cache hash table entries: 4096 (order: 2, 16384 bytes)
[    0.000000] Memory: 64MB = 64MB total
[    0.000000] Memory: 57880k/57880k available, 7656k reserved, 0K highmem
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
[    0.000000]     fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)
[    0.000000]     DMA     : 0xffc00000 - 0xffe00000   (   2 MB)
[    0.000000]     vmalloc : 0xc4800000 - 0xe0000000   ( 440 MB)
[    0.000000]     lowmem  : 0xc0000000 - 0xc4000000   (  64 MB)
[    0.000000]     modules : 0xbf000000 - 0xc0000000   (  16 MB)
[    0.000000]       .init : 0xc0008000 - 0xc0027000   ( 124 kB)
[    0.000000]       .text : 0xc0027000 - 0xc0486000   (4476 kB)
[    0.000000]       .data : 0xc04ca000 - 0xc06709c0   (1691 kB)
[    0.000000] SLUB: Genslabs=11, HWalign=32, Order=0-3, MinObjects=0,
CPUs=1, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  RCU-based detection of stalled CPUs is disabled.
[    0.000000]  Verbose stalled-CPUs detection is disabled.
[    0.000000] NR_IRQS:85
[    0.000000] irq: clearing pending status 02000000
[    0.000000] irq: clearing subpending status 00000007
[    0.000000] irq: clearing subpending status 00000002
[    0.000000] I-pipe, 8.437 MHz clocksource
[    0.000000] timer tcon=00500000, tcnt a4ca, tcfg 00000200,00000000,
usec 00001e57
[    0.000000] I-pipe 1.18-01: pipeline enabled.
[    0.000000] Console: colour dummy device 80x30
[    0.000000] console [ttySAC0] enabled
[    0.015000] Calibrating delay loop... 201.52 BogoMIPS (lpj=503808)
[    0.115000] pid_max: default: 32768 minimum: 301
[    0.115000] Mount-cache hash table entries: 512
[    0.120000] CPU: Testing write buffer coherency: ok
[    0.125000] devtmpfs: initialized
[    0.140000] gpiochip_add: gpios 288..303 (GPIOK) failed to register
[    0.140000] gpiochip_add: gpios 320..334 (GPIOL) failed to register
[    0.145000] gpiochip_add: gpios 352..353 (GPIOM) failed to register
[    0.160000] NET: Registered protocol family 16
[    0.220000] MINI2440: Option string mini2440=1tb
[    0.220000] MINI2440: LCD 0:240x320 [1:800x480] 2:1024x768
[    0.295000] S3C2440: Initialising architecture
[    0.295000] S3C2440: IRQ Support
[    0.300000] S3C24XX DMA Driver, Copyright 2003-2006 Simtec Electronics
[    0.305000] DMA channel 0 at c4808000, irq 33
[    0.310000] DMA channel 1 at c4808040, irq 34
[    0.315000] DMA channel 2 at c4808080, irq 35
[    0.320000] DMA channel 3 at c48080c0, irq 36
[    0.320000] S3C244X: Clock Support, DVS off
[    0.340000] s3c-adc s3c24xx-adc: attached adc driver
[    0.660000] bio: create slab <bio-0> at 0
[    0.685000] SCSI subsystem initialized
[    0.695000] usbcore: registered new interface driver usbfs
[    0.700000] usbcore: registered new interface driver hub
[    0.705000] usbcore: registered new device driver usb
[    0.715000] s3c-i2c s3c2440-i2c: slave address 0x10
[    0.715000] s3c-i2c s3c2440-i2c: bus frequency set to 98 KHz
[    0.730000] s3c-i2c s3c2440-i2c: i2c-0: S3C I2C adapter
[    0.755000] Advanced Linux Sound Architecture Driver Version 1.0.23.
[    0.765000] Switching to clocksource ipipe_tsc
(no further response, heartbeat LED doesn't blink)

Regards,
Alex

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Gilles Chanteperdrix]

Alexey Galakhov wrote:
> Hi,
> 
> I'm trying to run Xenomai on MINI2440 board. I use vanilla 2.6.35.9
> kernel patched with adeos-ipipe-2.6.35.9-arm-1.18-01.patch . In kernel
> configuration Xenomai is disabled, the only option (except debug) I
> turned on is CONFIG_IPIPE. It hangs.
> 
> I suspect it is somehow related to TIMER3 which is used along with
> TIMER4 on Samsung ARMs. Any suggestions?

There was a bug some time ago, which was fixed for TIMER3, see commit:
http://git.xenomai.org/?p=ipipe-gch.git;a=commit;h=ebf2fac4298e25509ba47942b9fad408dcefce1e

But I think your problem has more to do with the following commit, could
you try reverting it?
http://git.xenomai.org/?p=ipipe-gch.git;a=commitdiff;h=6ab29d9c7a4b119f45ef4d93780e894fe1c0c6c6;hp=cd9c5e016092258d4450e137be2d0844d0fe8b38

The same issue was reported on ixp4xx.

-- 
					    Gilles.

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Alexey Galakhov]

[-- Attachment #1: Type: text/plain, Size: 1726 bytes --]

>
> There was a bug some time ago, which was fixed for TIMER3, see commit:
>
> http://git.xenomai.org/?p=ipipe-gch.git;a=commit;h=ebf2fac4298e25509ba47942b9fad408dcefce1e
>

Oh, I know about this bug. That's the first thing I checked. I found that it
is already fixed in my case. However, it is still unclear for me where the
correct MUXes for timer4 and timer3 are initialized, seems to be kind of
magic. I didn't checked TCFGs against the datasheet, going to do it now. I
also think about making the free-running timer selectable from config since
some machines may have something wired to timer3's PWM.

I forgot to say, it works with patched kernel if CONFIG_IPIPE=no. Timers are
the same in both cases. I don't quite understand, however, how the
non-working free-running timer may affect the system in that case.
Interrupts from timer4 are still generated, so the system may boot with
ipipe off even with broken free-running timer without any obvious problems,
am I right?


> But I think your problem has more to do with the following commit, could
> you try reverting it?
>
> http://git.xenomai.org/?p=ipipe-gch.git;a=commitdiff;h=6ab29d9c7a4b119f45ef4d93780e894fe1c0c6c6;hp=cd9c5e016092258d4450e137be2d0844d0fe8b38
>
> The same issue was reported on ixp4xx.
>

I'll give it a try tomorrow, thanks. A little question about this part of
the patch:
-       if (!__ipipe_mach_timerstolen)
+       if (!__ipipe_mach_timerstolen) {
+               spin_lock(&timer_lock);
                getticksoffset_tscupdate();
+               spin_unlock(&timer_lock);
+       }
Is it a bugfix (forgotten spinlock added) or is it due to change of
getticksoffset_tscupdate() contents? (Just trying to understand how it
works.)

--
Alex

[-- Attachment #2: Type: text/html, Size: 2476 bytes --]

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Gilles Chanteperdrix]

Alexey Galakhov wrote:
>> There was a bug some time ago, which was fixed for TIMER3, see commit:
>>
>> http://git.xenomai.org/?p=ipipe-gch.git;a=commit;h=ebf2fac4298e25509ba47942b9fad408dcefce1e
>>
> 
> Oh, I know about this bug. That's the first thing I checked. I found that it
> is already fixed in my case. However, it is still unclear for me where the
> correct MUXes for timer4 and timer3 are initialized, seems to be kind of
> magic. I didn't checked TCFGs against the datasheet, going to do it now. I
> also think about making the free-running timer selectable from config since
> some machines may have something wired to timer3's PWM.
> 
> I forgot to say, it works with patched kernel if CONFIG_IPIPE=no. Timers are
> the same in both cases. I don't quite understand, however, how the
> non-working free-running timer may affect the system in that case.
> Interrupts from timer4 are still generated, so the system may boot with
> ipipe off even with broken free-running timer without any obvious problems,
> am I right?
> 
> 
>> But I think your problem has more to do with the following commit, could
>> you try reverting it?
>>
>> http://git.xenomai.org/?p=ipipe-gch.git;a=commitdiff;h=6ab29d9c7a4b119f45ef4d93780e894fe1c0c6c6;hp=cd9c5e016092258d4450e137be2d0844d0fe8b38
>>
>> The same issue was reported on ixp4xx.
>>
> 
> I'll give it a try tomorrow, thanks. A little question about this part of
> the patch:
> -       if (!__ipipe_mach_timerstolen)
> +       if (!__ipipe_mach_timerstolen) {
> +               spin_lock(&timer_lock);
>                 getticksoffset_tscupdate();
> +               spin_unlock(&timer_lock);
> +       }
> Is it a bugfix (forgotten spinlock added) or is it due to change of
> getticksoffset_tscupdate() contents? (Just trying to understand how it
> works.)

Looks like a mistake... Not having an S3C platform, I modified this code
and only compile-tested it.

-- 
                                                                Gilles.

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Gilles Chanteperdrix]

Alexey Galakhov wrote:
>> There was a bug some time ago, which was fixed for TIMER3, see commit:
>>
>> http://git.xenomai.org/?p=ipipe-gch.git;a=commit;h=ebf2fac4298e25509ba47942b9fad408dcefce1e
>>
> 
> Oh, I know about this bug. That's the first thing I checked. I found that it
> is already fixed in my case. However, it is still unclear for me where the
> correct MUXes for timer4 and timer3 are initialized, seems to be kind of
> magic. I didn't checked TCFGs against the datasheet, going to do it now. I
> also think about making the free-running timer selectable from config since
> some machines may have something wired to timer3's PWM.
> 
> I forgot to say, it works with patched kernel if CONFIG_IPIPE=no. Timers are
> the same in both cases. I don't quite understand, however, how the
> non-working free-running timer may affect the system in that case.
> Interrupts from timer4 are still generated, so the system may boot with
> ipipe off even with broken free-running timer without any obvious problems,
> am I right?

I have absolutely no idea. Fabian Godehart sent the patch, saying that
it fixed an issue, I merged it. That is all.


-- 
                                                                Gilles.

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Alexey Galakhov]

On 04/27/2011 09:56 PM, Gilles Chanteperdrix wrote:
> But I think your problem has more to do with the following commit, could
> you try reverting it?
> http://git.xenomai.org/?p=ipipe-gch.git;a=commitdiff;h=6ab29d9c7a4b119f45ef4d93780e894fe1c0c6c6;hp=cd9c5e016092258d4450e137be2d0844d0fe8b38
>
> The same issue was reported on ixp4xx.
Just tried to revert this patch. Hangs. But now in a different way: it
starts to initialize peripherals and stops on LED initialization. Looks
like race conditions since it stops in slightly different place each
time I try to boot (randomly at led3, led4 or "backlight" LED). Last
lines of dmesg are, i.e.:

[    3.090000] samsung-ts s3c2440-ts: driver attached, registering input
device
[    3.100000] input: S3C24XX TouchScreen as /devices/virtual/input/input0
[    3.120000] S3C24XX RTC, (c) 2004,2006 Simtec Electronics
[    3.120000] s3c-rtc s3c2410-rtc: rtc disabled, re-enabling
[    3.135000] s3c-rtc s3c2410-rtc: rtc core: registered s3c as rtc0
[    3.135000] i2c /dev entries driver
[    3.155000] S3C2410 Watchdog Timer, (c) 2004 Simtec Electronics
[    3.160000] s3c2410-wdt s3c2410-wdt: watchdog inactive, reset
disabled, irq enabled
[    3.180000] s3c-sdi s3c2440-sdi: powered down.
[    3.180000] s3c-sdi s3c2440-sdi: mmc0 - using pio, sw SDIO IRQ
[    3.185000] s3c-sdi s3c2440-sdi: running at 0kHz (requested: 0kHz).
[    3.200000] Registered led device: led1
[    3.205000] Registered led device: led2
[    3.210000] s3c-sdi s3c2440-sdi: running at 398kHz (requested: 400kHz).
[    3.215000] Registered led device: led3
[    3.220000] Registered led device: led4
(hangs)

Going to try kgdb over serial line.

--
Alex

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Gilles Chanteperdrix]

Alexey Galakhov wrote:
> On 04/27/2011 09:56 PM, Gilles Chanteperdrix wrote:
>> But I think your problem has more to do with the following commit, could
>> you try reverting it?
>> http://git.xenomai.org/?p=ipipe-gch.git;a=commitdiff;h=6ab29d9c7a4b119f45ef4d93780e894fe1c0c6c6;hp=cd9c5e016092258d4450e137be2d0844d0fe8b38
>>
>> The same issue was reported on ixp4xx.
> Just tried to revert this patch. Hangs. But now in a different way: it
> starts to initialize peripherals and stops on LED initialization. Looks
> like race conditions since it stops in slightly different place each
> time I try to boot (randomly at led3, led4 or "backlight" LED). Last
> lines of dmesg are, i.e.:
> 
> [    3.090000] samsung-ts s3c2440-ts: driver attached, registering input
> device
> [    3.100000] input: S3C24XX TouchScreen as /devices/virtual/input/input0
> [    3.120000] S3C24XX RTC, (c) 2004,2006 Simtec Electronics
> [    3.120000] s3c-rtc s3c2410-rtc: rtc disabled, re-enabling
> [    3.135000] s3c-rtc s3c2410-rtc: rtc core: registered s3c as rtc0
> [    3.135000] i2c /dev entries driver
> [    3.155000] S3C2410 Watchdog Timer, (c) 2004 Simtec Electronics
> [    3.160000] s3c2410-wdt s3c2410-wdt: watchdog inactive, reset
> disabled, irq enabled
> [    3.180000] s3c-sdi s3c2440-sdi: powered down.
> [    3.180000] s3c-sdi s3c2440-sdi: mmc0 - using pio, sw SDIO IRQ
> [    3.185000] s3c-sdi s3c2440-sdi: running at 0kHz (requested: 0kHz).
> [    3.200000] Registered led device: led1
> [    3.205000] Registered led device: led2
> [    3.210000] s3c-sdi s3c2440-sdi: running at 398kHz (requested: 400kHz).
> [    3.215000] Registered led device: led3
> [    3.220000] Registered led device: led4
> (hangs)
> 
> Going to try kgdb over serial line.

Ok. Better try without this patch first, then try reapplying the patch
when you are certain that everything else works correctly. The patch
enables some assembly code that was never actually tested. On my side, I
will try and test it separately.

-- 
                                                                Gilles.

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Alexey Galakhov]

On 04/28/2011 05:16 PM, Gilles Chanteperdrix wrote:
> Alexey Galakhov wrote:
>> Going to try kgdb over serial line.
> Ok. Better try without this patch first, then try reapplying the patch
> when you are certain that everything else works correctly. The patch
> enables some assembly code that was never actually tested. On my side, I
> will try and test it separately.
That's exactly what I want to do.

Is there any documentation on ipipe's internals involved in time.c ?
Reading C code is certainly not the fastest way to understand how things
work :)

--
Alex

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Gilles Chanteperdrix]

Alexey Galakhov wrote:
> On 04/28/2011 05:16 PM, Gilles Chanteperdrix wrote:
>> Alexey Galakhov wrote:
>>> Going to try kgdb over serial line.
>> Ok. Better try without this patch first, then try reapplying the patch
>> when you are certain that everything else works correctly. The patch
>> enables some assembly code that was never actually tested. On my side, I
>> will try and test it separately.
> That's exactly what I want to do.
> 
> Is there any documentation on ipipe's internals involved in time.c ?
> Reading C code is certainly not the fastest way to understand how things
> work :)

http://www.xenomai.org/index.php/I-pipe:ArmPorting

The parts about the tsc and gpio demuxing are outdated. But the part
about the timer is still correct.

-- 
                                                                Gilles.

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Alexey Galakhov]

On 04/28/2011 05:16 PM, Gilles Chanteperdrix wrote:
> Ok. Better try without this patch first, then try reapplying the patch
> when you are certain that everything else works correctly. The patch
> enables some assembly code that was never actually tested. On my side, I
> will try and test it separately.
>
I just got a couple of stack traces via JTAG (just stopped and restarted
the "hanging" processor multiple times and doing backtrace). It runs all
the time somewhere inside __ipipe_handle_irq or __ipipe_grab_irq():

#0  0xc00281b8 in __gnu_mcount_nc ()
#1  0xc002f7ec in __ipipe_handle_irq ()
#2  0xc002fb40 in __ipipe_grab_irq ()
#3  0xc0027b8c in __irq_svc ()
#4  0xc0027b8c in __irq_svc ()
#5  0xc0027b8c in __irq_svc ()
(a lot of __irq_svc() follows)

#0  0xc0092ba4 in __ipipe_trace ()
#1  0xc00937d4 in ipipe_trace_begin ()
#2  0xc002fb34 in __ipipe_grab_irq ()
#3  0xc0027b8c in __irq_svc ()
#4  0xc0027b8c in __irq_svc ()
(etc)

#0  0xc002fae0 in __ipipe_grab_irq ()
#1  0xc0027b8c in __irq_svc ()
#2  0xc0027b8c in __irq_svc ()
(etc)

#0  0xc0092948 in __ipipe_trace ()
#1  0xc0093448 in ipipe_trace_function ()
#2  0xc0094a90 in ftrace_test_stop_func ()
#3  0xc00281ec in gnu_trace ()
#4  0xc00281ec in gnu_trace ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

and so on. Hope that helps.

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Gilles Chanteperdrix]

Alexey Galakhov wrote:
> On 04/28/2011 05:16 PM, Gilles Chanteperdrix wrote:
>> Ok. Better try without this patch first, then try reapplying the patch
>> when you are certain that everything else works correctly. The patch
>> enables some assembly code that was never actually tested. On my side, I
>> will try and test it separately.
>>
> I just got a couple of stack traces via JTAG (just stopped and restarted
> the "hanging" processor multiple times and doing backtrace). It runs all
> the time somewhere inside __ipipe_handle_irq or __ipipe_grab_irq():
> 
> #0  0xc00281b8 in __gnu_mcount_nc ()
> #1  0xc002f7ec in __ipipe_handle_irq ()
> #2  0xc002fb40 in __ipipe_grab_irq ()
> #3  0xc0027b8c in __irq_svc ()
> #4  0xc0027b8c in __irq_svc ()
> #5  0xc0027b8c in __irq_svc ()
> (a lot of __irq_svc() follows)
> 
> #0  0xc0092ba4 in __ipipe_trace ()
> #1  0xc00937d4 in ipipe_trace_begin ()
> #2  0xc002fb34 in __ipipe_grab_irq ()
> #3  0xc0027b8c in __irq_svc ()
> #4  0xc0027b8c in __irq_svc ()
> (etc)
> 
> #0  0xc002fae0 in __ipipe_grab_irq ()
> #1  0xc0027b8c in __irq_svc ()
> #2  0xc0027b8c in __irq_svc ()
> (etc)
> 
> #0  0xc0092948 in __ipipe_trace ()
> #1  0xc0093448 in ipipe_trace_function ()
> #2  0xc0094a90 in ftrace_test_stop_func ()
> #3  0xc00281ec in gnu_trace ()
> #4  0xc00281ec in gnu_trace ()
> Backtrace stopped: previous frame identical to this frame (corrupt stack?)
> 
> and so on. Hope that helps.
> 

Try disabling the I-pipe tracer.

-- 
					    Gilles.

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Alexey Galakhov]

On 04/28/2011 05:16 PM, Gilles Chanteperdrix wrote:
> Ok. Better try without this patch first, then try reapplying the patch
> when you are certain that everything else works correctly. The patch
> enables some assembly code that was never actually tested. On my side, I
> will try and test it separately.
So I tried to trace the kernel using gdb and jtag.

It enters the interrupt handler over and over again. If I set a
breakpoint at __ipipe_grab_irq, I see the following:
__ipipe_grab_irq (irq=37, regs=0xc3dd3f08)
__ipipe_grab_irq (irq=30, regs=0xc3dd3f08)
__ipipe_grab_irq (irq=37, regs=0xc3dd3f08)
__ipipe_grab_irq (irq=30, regs=0xc3dd3f08)
and so on. Note the strict order of IRQs 30 and 37.

According to /proc/interrupts running vanilla kernel, these IRQs are:
 30:      36105         s3c  S3C2410 Timer Tick
 37:      12464         s3c  s3c-mci

As far as I can see, these two IRQs are both handled by the same code in
ipipe. And they are both periodic and occur so frequently that they
interrupt each other's handler. Looks like they aren't masked properly
for some reason. So the kernel loops infinitely in these two interrupt
handlers and does never exit them!

--
Alex

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Gilles Chanteperdrix]

Alexey Galakhov wrote:
> On 04/28/2011 05:16 PM, Gilles Chanteperdrix wrote:
>> Ok. Better try without this patch first, then try reapplying the patch
>> when you are certain that everything else works correctly. The patch
>> enables some assembly code that was never actually tested. On my side, I
>> will try and test it separately.
> So I tried to trace the kernel using gdb and jtag.
> 
> It enters the interrupt handler over and over again. If I set a
> breakpoint at __ipipe_grab_irq, I see the following:
> __ipipe_grab_irq (irq=37, regs=0xc3dd3f08)
> __ipipe_grab_irq (irq=30, regs=0xc3dd3f08)
> __ipipe_grab_irq (irq=37, regs=0xc3dd3f08)
> __ipipe_grab_irq (irq=30, regs=0xc3dd3f08)
> and so on. Note the strict order of IRQs 30 and 37.
> 
> According to /proc/interrupts running vanilla kernel, these IRQs are:
>  30:      36105         s3c  S3C2410 Timer Tick
>  37:      12464         s3c  s3c-mci
> 
> As far as I can see, these two IRQs are both handled by the same code in
> ipipe. And they are both periodic and occur so frequently that they
> interrupt each other's handler. Looks like they aren't masked properly
> for some reason. So the kernel loops infinitely in these two interrupt
> handlers and does never exit them!

Well, is not that an effect of kgdb? The timer tick occurs 100 or 1000
times per second, so, you can not keep up with gdb. The fact that a new
interrupts may occur why handling another interrupt is normal with Adeos.

It is not to say that there is no irq loop, but we can not conclude from
the symptoms you are describing.

Is there an earlier version of the I-pipe patch which was working?



-- 
					    Gilles.

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Alexey Galakhov]

[-- Attachment #1: Type: text/plain, Size: 1324 bytes --]

2011/4/28 Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>

>
> Well, is not that an effect of kgdb? The timer tick occurs 100 or 1000
> times per second, so, you can not keep up with gdb. The fact that a new
> interrupts may occur why handling another interrupt is normal with Adeos.
>
> It is not to say that there is no irq loop, but we can not conclude from
> the symptoms you are describing.
>
>
I understand this possibility. I did a simple test: just switched the
processor back and forth between "run" and "halt" many many times. I did it
on hardware (JTAG) level so I believe the CPU was running at full speed. And
it was catched in one of these interrupt handlers every time, I've never
seen it in any other function. Thus I believe it was doing nothing else. The
stack is suspicious as well: it has lots of IRQ stuff, nothing but IRQs,
even if I interrupt the CPU for the first time.



> Is there an earlier version of the I-pipe patch which was working?
>
>
Unfortunately I've not tested earlier versions. I'll try. They say it worked
with linux 2.6.29 (ipipe version not mentioned, but it was just before the
mini2440 timer3 patch was submitted to mainstream):
http://www.friendlyarm.net/forum/topic/598

Seems that these guys have the same problem:
http://www.friendlyarm.net/forum/topic/1108

--
Alex

[-- Attachment #2: Type: text/html, Size: 1997 bytes --]

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Gilles Chanteperdrix]

Alexey Galakhov wrote:
> 2011/4/28 Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> 
>> Well, is not that an effect of kgdb? The timer tick occurs 100 or 1000
>> times per second, so, you can not keep up with gdb. The fact that a new
>> interrupts may occur why handling another interrupt is normal with Adeos.
>>
>> It is not to say that there is no irq loop, but we can not conclude from
>> the symptoms you are describing.
>>
>>
> I understand this possibility. I did a simple test: just switched the
> processor back and forth between "run" and "halt" many many times. I did it
> on hardware (JTAG) level so I believe the CPU was running at full speed. And
> it was catched in one of these interrupt handlers every time, I've never
> seen it in any other function. Thus I believe it was doing nothing else. The
> stack is suspicious as well: it has lots of IRQ stuff, nothing but IRQs,
> even if I interrupt the CPU for the first time.
> 
> 
> 
>> Is there an earlier version of the I-pipe patch which was working?
>>
>>
> Unfortunately I've not tested earlier versions. I'll try. They say it worked
> with linux 2.6.29 (ipipe version not mentioned, but it was just before the
> mini2440 timer3 patch was submitted to mainstream):
> http://www.friendlyarm.net/forum/topic/598
> 
> Seems that these guys have the same problem:
> http://www.friendlyarm.net/forum/topic/1108

So, have you tried reverting the timer3 patch?

-- 
                                                                Gilles.

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Alexey Galakhov]

[-- Attachment #1: Type: text/plain, Size: 555 bytes --]

2011/4/29 Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>

>
>
> So, have you tried reverting the timer3 patch?
>
>
Yes. After reverting, EVERYTHING stops working correctly (even with ipipe
off). It is really necessary.

Now I'm trying to understand the MMC code. Looks like that timers do their
job correctly, and MMC/SD driver seems to be the source of irq flood. If so,
it depends on kernel version, not on ipipe version. That is, ipipe seems
just to trigger a bug in already buggy kernel. Trying to examine it more
closely. Not sure.

--
Alex

[-- Attachment #2: Type: text/html, Size: 896 bytes --]

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Gilles Chanteperdrix]

Alexey Galakhov wrote:
> 2011/4/29 Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> 
>>
>> So, have you tried reverting the timer3 patch?
>>
>>
> Yes. After reverting, EVERYTHING stops working correctly (even with ipipe
> off). It is really necessary.
> 
> Now I'm trying to understand the MMC code. Looks like that timers do their
> job correctly, and MMC/SD driver seems to be the source of irq flood. If so,
> it depends on kernel version, not on ipipe version. That is, ipipe seems
> just to trigger a bug in already buggy kernel. Trying to examine it more
> closely. Not sure.

Maybe you can try booting with MMC disabled, to see if the I-pipe is
working? Not permanently, just to confirm that you are not chasing the
wrong bug.

-- 
                                                                Gilles.

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Alexey Galakhov]

[-- Attachment #1: Type: text/plain, Size: 564 bytes --]

2011/4/29 Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>

>
> Maybe you can try booting with MMC disabled, to see if the I-pipe is
> working? Not permanently, just to confirm that you are not chasing the
> wrong bug.
>

Sure, that's exactly what I want to do. Unfortunately, I cannot boot to a
working system in that mode since I have root on SD card, but "Kernel panic
- unable to mount root fs" would indicate successful boot. I think also
about moving root somewhere else to be able to run tests ans examine /proc
and /sys under ipipe kernel.

--
Alex

[-- Attachment #2: Type: text/html, Size: 920 bytes --]

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Gilles Chanteperdrix]

Alexey Galakhov wrote:
> 2011/4/29 Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> 
>> Maybe you can try booting with MMC disabled, to see if the I-pipe is
>> working? Not permanently, just to confirm that you are not chasing the
>> wrong bug.
>>
> 
> Sure, that's exactly what I want to do. Unfortunately, I cannot boot to a
> working system in that mode since I have root on SD card, but "Kernel panic
> - unable to mount root fs" would indicate successful boot. I think also
> about moving root somewhere else to be able to run tests ans examine /proc
> and /sys under ipipe kernel.

Just made a rootfs for s3c, you can find it as a tar archive here:
http://xenomai.org/~gch/rootfs-s3c.tar.bz2

or as a cpio archive to be used as an initramfs here:
http://xenomai.org/~gch/initramfs-s3c.tar.bz2


-- 
                                                                Gilles.

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Gilles Chanteperdrix]

Gilles Chanteperdrix wrote:
> Alexey Galakhov wrote:
>> 2011/4/29 Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
>>
>>> Maybe you can try booting with MMC disabled, to see if the I-pipe is
>>> working? Not permanently, just to confirm that you are not chasing the
>>> wrong bug.
>>>
>> Sure, that's exactly what I want to do. Unfortunately, I cannot boot to a
>> working system in that mode since I have root on SD card, but "Kernel panic
>> - unable to mount root fs" would indicate successful boot. I think also
>> about moving root somewhere else to be able to run tests ans examine /proc
>> and /sys under ipipe kernel.
> 
> Just made a rootfs for s3c, you can find it as a tar archive here:
> http://xenomai.org/~gch/rootfs-s3c.tar.bz2
> 
> or as a cpio archive to be used as an initramfs here:
> http://xenomai.org/~gch/initramfs-s3c.tar.bz2

s/.bz2/.gz/

-- 
                                                                Gilles.

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Alexey Galakhov]

On 04/29/2011 01:32 AM, Gilles Chanteperdrix wrote:
> Maybe you can try booting with MMC disabled, to see if the I-pipe is
> working? Not permanently, just to confirm that you are not chasing the
> wrong bug.
Thank you for the root image!

Just did some simple checks.

It works with the same kernel, just with MMC card removed (heartbeat
blinks, "Waiting for root").
It STOPS working (heartbeat stops blinking) immediately after inserting
either MMC or USB flash. So it is not exactly MMC related, any interrupt
source (both MMC and USB) results in lockup.

The lockup is not immediate, it works for some short time (enough to
print kind of "new device found" to dmesg).

It does not depend on drivers actually compiled in, it depends on
devices being initialized. It stops working as soon as some hardware
starts sending frequent interrupts.

Now going to try your asm patch.

--
Alex

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Gilles Chanteperdrix]

Alexey Galakhov wrote:
> On 04/29/2011 01:32 AM, Gilles Chanteperdrix wrote:
>> Maybe you can try booting with MMC disabled, to see if the I-pipe is
>> working? Not permanently, just to confirm that you are not chasing the
>> wrong bug.
> Thank you for the root image!
> 
> Just did some simple checks.
> 
> It works with the same kernel, just with MMC card removed (heartbeat
> blinks, "Waiting for root").
> It STOPS working (heartbeat stops blinking) immediately after inserting
> either MMC or USB flash. So it is not exactly MMC related, any interrupt
> source (both MMC and USB) results in lockup.
> 
> The lockup is not immediate, it works for some short time (enough to
> print kind of "new device found" to dmesg).
> 
> It does not depend on drivers actually compiled in, it depends on
> devices being initialized. It stops working as soon as some hardware
> starts sending frequent interrupts.
> 
> Now going to try your asm patch.

Ok. Two things to check:
- if the irqs are handled by handle_edge, try using handle_level instead;
- if the irqs are demuxed gpios, check that ipipe_handle_chained_irq is
used instead of generic_handle_irq.

I will check on my side tonight.

-- 
					    Gilles.

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Alexey Galakhov]

On 04/29/2011 05:32 PM, Gilles Chanteperdrix wrote:
> Alexey Galakhov wrote:
> Now going to try your asm patch.
> Ok. Two things to check:
> - if the irqs are handled by handle_edge, try using handle_level instead;
> - if the irqs are demuxed gpios, check that ipipe_handle_chained_irq is
> used instead of generic_handle_irq.
>
> I will check on my side tonight.
Same with the patch.

--
Alex

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Gilles Chanteperdrix]

Alexey Galakhov wrote:
> On 04/29/2011 05:32 PM, Gilles Chanteperdrix wrote:
>> Alexey Galakhov wrote:
>> Now going to try your asm patch.
>> Ok. Two things to check:
>> - if the irqs are handled by handle_edge, try using handle_level instead;
>> - if the irqs are demuxed gpios, check that ipipe_handle_chained_irq is
>> used instead of generic_handle_irq.
>>
>> I will check on my side tonight.
> Same with the patch.

The patch was supposed to fix the hang at "Switching to ipipe_tsc
source". Does your kernel still hang there?

-- 
					    Gilles.

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Alexey Galakhov]

On 04/29/2011 06:14 PM, Gilles Chanteperdrix wrote:
> Alexey Galakhov wrote:
>> On 04/29/2011 05:32 PM, Gilles Chanteperdrix wrote:
>>> Alexey Galakhov wrote:
>>> Now going to try your asm patch.
>>> Ok. Two things to check:
>>> - if the irqs are handled by handle_edge, try using handle_level instead;
>>> - if the irqs are demuxed gpios, check that ipipe_handle_chained_irq is
>>> used instead of generic_handle_irq.
>>>
>>> I will check on my side tonight.
>> Same with the patch.
> The patch was supposed to fix the hang at "Switching to ipipe_tsc
> source". Does your kernel still hang there?
>
Sorry, my mistake, used an incorrect uImage to boot.

The "ipipe_tsc clocksource" bug is FIXED now. Thank you very much!

Now chasing the first bug, mmc interrupt flood...

--
Alex

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Gilles Chanteperdrix]

Gilles Chanteperdrix wrote:
> Alexey Galakhov wrote:
>> On 04/29/2011 01:32 AM, Gilles Chanteperdrix wrote:
>>> Maybe you can try booting with MMC disabled, to see if the I-pipe is
>>> working? Not permanently, just to confirm that you are not chasing the
>>> wrong bug.
>> Thank you for the root image!
>>
>> Just did some simple checks.
>>
>> It works with the same kernel, just with MMC card removed (heartbeat
>> blinks, "Waiting for root").
>> It STOPS working (heartbeat stops blinking) immediately after inserting
>> either MMC or USB flash. So it is not exactly MMC related, any interrupt
>> source (both MMC and USB) results in lockup.
>>
>> The lockup is not immediate, it works for some short time (enough to
>> print kind of "new device found" to dmesg).
>>
>> It does not depend on drivers actually compiled in, it depends on
>> devices being initialized. It stops working as soon as some hardware
>> starts sending frequent interrupts.
>>
>> Now going to try your asm patch.
> 
> Ok. Two things to check:
> - if the irqs are handled by handle_edge, try using handle_level instead;
> - if the irqs are demuxed gpios, check that ipipe_handle_chained_irq is
> used instead of generic_handle_irq.
> 
> I will check on my side tonight.

Please try the following patch:

diff --git a/arch/arm/plat-s3c24xx/irq.c b/arch/arm/plat-s3c24xx/irq.c
index c48d99a..8a29d9d 100644
--- a/arch/arm/plat-s3c24xx/irq.c
+++ b/arch/arm/plat-s3c24xx/irq.c
@@ -627,7 +627,7 @@ void __init s3c24xx_init_irq(void)
 		default:
 			//irqdbf("registering irq %d (s3c irq)\n", irqno);
 			set_irq_chip(irqno, &s3c_irq_chip);
-			set_irq_handler(irqno, handle_edge_irq);
+			set_irq_handler(irqno, handle_level_irq);
 			set_irq_flags(irqno, IRQF_VALID);
 		}
 	}
@@ -647,14 +647,14 @@ void __init s3c24xx_init_irq(void)
 	for (irqno = IRQ_EINT0; irqno <= IRQ_EINT3; irqno++) {
 		irqdbf("registering irq %d (ext int)\n", irqno);
 		set_irq_chip(irqno, &s3c_irq_eint0t4);
-		set_irq_handler(irqno, handle_edge_irq);
+		set_irq_handler(irqno, handle_level_irq);
 		set_irq_flags(irqno, IRQF_VALID);
 	}
 
 	for (irqno = IRQ_EINT4; irqno <= IRQ_EINT23; irqno++) {
 		irqdbf("registering irq %d (extended s3c irq)\n", irqno);
 		set_irq_chip(irqno, &s3c_irqext_chip);
-		set_irq_handler(irqno, handle_edge_irq);
+		set_irq_handler(irqno, handle_level_irq);
 		set_irq_flags(irqno, IRQF_VALID);
 	}
 
@@ -686,7 +686,7 @@ void __init s3c24xx_init_irq(void)
 	for (irqno = IRQ_TC; irqno <= IRQ_ADC; irqno++) {
 		irqdbf("registering irq %d (s3c adc irq)\n", irqno);
 		set_irq_chip(irqno, &s3c_irq_adc);
-		set_irq_handler(irqno, handle_edge_irq);
+		set_irq_handler(irqno, handle_level_irq);
 		set_irq_flags(irqno, IRQF_VALID);
 	}
 
diff --git a/arch/arm/plat-samsung/irq-uart.c b/arch/arm/plat-samsung/irq-uart.c
index 4f8c102..1da5aff 100644
--- a/arch/arm/plat-samsung/irq-uart.c
+++ b/arch/arm/plat-samsung/irq-uart.c
@@ -88,13 +88,13 @@ static void s3c_irq_demux_uart(unsigned int irq, struct irq_desc *desc)
 	int base = uirq->base_irq;
 
 	if (pend & (1 << 0))
-		generic_handle_irq(base);
+		ipipe_handle_chained_irq(base);
 	if (pend & (1 << 1))
-		generic_handle_irq(base + 1);
+		ipipe_handle_chained_irq(base + 1);
 	if (pend & (1 << 2))
-		generic_handle_irq(base + 2);
+		ipipe_handle_chained_irq(base + 2);
 	if (pend & (1 << 3))
-		generic_handle_irq(base + 3);
+		ipipe_handle_chained_irq(base + 3);
 }
 
 static struct irq_chip s3c_irq_uart = {


-- 
                                                                Gilles.

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Alexey Galakhov]

[-- Attachment #1: Type: text/plain, Size: 2894 bytes --]

2011/4/30 Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>

>
> Please try the following patch:
>
> diff --git a/arch/arm/plat-s3c24xx/irq.c b/arch/arm/plat-s3c24xx/irq.c
> index c48d99a..8a29d9d 100644
> --- a/arch/arm/plat-s3c24xx/irq.c
> +++ b/arch/arm/plat-s3c24xx/irq.c
> @@ -627,7 +627,7 @@ void __init s3c24xx_init_irq(void)
>                default:
>                        //irqdbf("registering irq %d (s3c irq)\n", irqno);
>                        set_irq_chip(irqno, &s3c_irq_chip);
> -                       set_irq_handler(irqno, handle_edge_irq);
> +                       set_irq_handler(irqno, handle_level_irq);
>                        set_irq_flags(irqno, IRQF_VALID);
>                }
>        }
> @@ -647,14 +647,14 @@ void __init s3c24xx_init_irq(void)
>        for (irqno = IRQ_EINT0; irqno <= IRQ_EINT3; irqno++) {
>                irqdbf("registering irq %d (ext int)\n", irqno);
>                set_irq_chip(irqno, &s3c_irq_eint0t4);
> -               set_irq_handler(irqno, handle_edge_irq);
> +               set_irq_handler(irqno, handle_level_irq);
>                set_irq_flags(irqno, IRQF_VALID);
>        }
>
>        for (irqno = IRQ_EINT4; irqno <= IRQ_EINT23; irqno++) {
>                irqdbf("registering irq %d (extended s3c irq)\n", irqno);
>                set_irq_chip(irqno, &s3c_irqext_chip);
> -               set_irq_handler(irqno, handle_edge_irq);
> +               set_irq_handler(irqno, handle_level_irq);
>                set_irq_flags(irqno, IRQF_VALID);
>        }
>
> @@ -686,7 +686,7 @@ void __init s3c24xx_init_irq(void)
>        for (irqno = IRQ_TC; irqno <= IRQ_ADC; irqno++) {
>                irqdbf("registering irq %d (s3c adc irq)\n", irqno);
>                set_irq_chip(irqno, &s3c_irq_adc);
> -               set_irq_handler(irqno, handle_edge_irq);
> +               set_irq_handler(irqno, handle_level_irq);
>                set_irq_flags(irqno, IRQF_VALID);
>        }
>
> diff --git a/arch/arm/plat-samsung/irq-uart.c
> b/arch/arm/plat-samsung/irq-uart.c
> index 4f8c102..1da5aff 100644
> --- a/arch/arm/plat-samsung/irq-uart.c
> +++ b/arch/arm/plat-samsung/irq-uart.c
> @@ -88,13 +88,13 @@ static void s3c_irq_demux_uart(unsigned int irq, struct
> irq_desc *desc)
>        int base = uirq->base_irq;
>
>        if (pend & (1 << 0))
> -               generic_handle_irq(base);
> +               ipipe_handle_chained_irq(base);
>        if (pend & (1 << 1))
> -               generic_handle_irq(base + 1);
> +               ipipe_handle_chained_irq(base + 1);
>        if (pend & (1 << 2))
> -               generic_handle_irq(base + 2);
> +               ipipe_handle_chained_irq(base + 2);
>        if (pend & (1 << 3))
> -               generic_handle_irq(base + 3);
> +               ipipe_handle_chained_irq(base + 3);
>  }
>
>  static struct irq_chip s3c_irq_uart = {
>
>
> It works!! Thank you!

--
Alex

[-- Attachment #2: Type: text/html, Size: 3554 bytes --]

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Alexey Galakhov]

[-- Attachment #1: Type: text/plain, Size: 198 bytes --]

2011/4/30 Alexey Galakhov <agalakhov@domain.hid>

>
> It works!! Thank you!
>
> --
> Alex
>
>
Hmmmm... It seems to be unstable, it generally works but with random
hangups. Trying to debug.

--
Alex

[-- Attachment #2: Type: text/html, Size: 531 bytes --]

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Alexey Galakhov]

[-- Attachment #1: Type: text/plain, Size: 300 bytes --]

2011/4/30 Alexey Galakhov <agalakhov@domain.hid>

>
> Hmmmm... It seems to be unstable, it generally works but with random
> hangups. Trying to debug.
>

Looks like now it's not IRQ related. Seems to be kind of overflow in
logarithmic_accumulation (), probably due to changes in timer code.

--
Alex

[-- Attachment #2: Type: text/html, Size: 586 bytes --]

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Gilles Chanteperdrix]

Alexey Galakhov wrote:
> 2011/4/30 Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> 
>> Please try the following patch:
>>
>> diff --git a/arch/arm/plat-s3c24xx/irq.c b/arch/arm/plat-s3c24xx/irq.c
>> index c48d99a..8a29d9d 100644
>> --- a/arch/arm/plat-s3c24xx/irq.c
>> +++ b/arch/arm/plat-s3c24xx/irq.c
>> @@ -627,7 +627,7 @@ void __init s3c24xx_init_irq(void)
>>                default:
>>                        //irqdbf("registering irq %d (s3c irq)\n", irqno);
>>                        set_irq_chip(irqno, &s3c_irq_chip);
>> -                       set_irq_handler(irqno, handle_edge_irq);
>> +                       set_irq_handler(irqno, handle_level_irq);
>>                        set_irq_flags(irqno, IRQF_VALID);
>>                }
>>        }
>> @@ -647,14 +647,14 @@ void __init s3c24xx_init_irq(void)
>>        for (irqno = IRQ_EINT0; irqno <= IRQ_EINT3; irqno++) {
>>                irqdbf("registering irq %d (ext int)\n", irqno);
>>                set_irq_chip(irqno, &s3c_irq_eint0t4);
>> -               set_irq_handler(irqno, handle_edge_irq);
>> +               set_irq_handler(irqno, handle_level_irq);
>>                set_irq_flags(irqno, IRQF_VALID);
>>        }
>>
>>        for (irqno = IRQ_EINT4; irqno <= IRQ_EINT23; irqno++) {
>>                irqdbf("registering irq %d (extended s3c irq)\n", irqno);
>>                set_irq_chip(irqno, &s3c_irqext_chip);
>> -               set_irq_handler(irqno, handle_edge_irq);
>> +               set_irq_handler(irqno, handle_level_irq);
>>                set_irq_flags(irqno, IRQF_VALID);
>>        }
>>
>> @@ -686,7 +686,7 @@ void __init s3c24xx_init_irq(void)
>>        for (irqno = IRQ_TC; irqno <= IRQ_ADC; irqno++) {
>>                irqdbf("registering irq %d (s3c adc irq)\n", irqno);
>>                set_irq_chip(irqno, &s3c_irq_adc);
>> -               set_irq_handler(irqno, handle_edge_irq);
>> +               set_irq_handler(irqno, handle_level_irq);
>>                set_irq_flags(irqno, IRQF_VALID);
>>        }
>>
>> diff --git a/arch/arm/plat-samsung/irq-uart.c
>> b/arch/arm/plat-samsung/irq-uart.c
>> index 4f8c102..1da5aff 100644
>> --- a/arch/arm/plat-samsung/irq-uart.c
>> +++ b/arch/arm/plat-samsung/irq-uart.c
>> @@ -88,13 +88,13 @@ static void s3c_irq_demux_uart(unsigned int irq, struct
>> irq_desc *desc)
>>        int base = uirq->base_irq;
>>
>>        if (pend & (1 << 0))
>> -               generic_handle_irq(base);
>> +               ipipe_handle_chained_irq(base);
>>        if (pend & (1 << 1))
>> -               generic_handle_irq(base + 1);
>> +               ipipe_handle_chained_irq(base + 1);
>>        if (pend & (1 << 2))
>> -               generic_handle_irq(base + 2);
>> +               ipipe_handle_chained_irq(base + 2);
>>        if (pend & (1 << 3))
>> -               generic_handle_irq(base + 3);
>> +               ipipe_handle_chained_irq(base + 3);
>>  }
>>
>>  static struct irq_chip s3c_irq_uart = {
>>
>>
>> It works!! Thank you!

Ok. Could you try only keeping the part with ipipe_handle_chained_irq,
leaving the handle_edge_irqs?

-- 
                                                                Gilles.

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Alexey Galakhov]

[-- Attachment #1: Type: text/plain, Size: 1335 bytes --]

2011/5/1 Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>

>
> Ok. Could you try only keeping the part with ipipe_handle_chained_irq,
> leaving the handle_edge_irqs?
>

Sure. I'll do it tomorrow.

There is another one issue: the board hangs after about 2-3 minutes running.
I found that it is caused by a semi-permanent (order of 2^64 iterations)
loop in (untouched) kernel timer code, most likely due to weird return value
of __ipipe_tsc_get. I haven't found the source of this problem yet. I tried
a small fix:

--- time.c.org    2011-04-26 20:52:00.000000000 +0600
+++ time.c    2011-05-01 02:03:33.000000000 +0600
@@ -124,7 +124,7 @@

 static inline unsigned long timer_freerunning_getvalue(void)
 {
-    return __raw_readl(S3C2410_TCNTO(3));
+    return (unsigned long)__raw_readw(S3C2410_TCNTO(3));
 }

 static inline unsigned long timer_freerunning_getticksoffset(unsigned long
tval)
@@ -390,6 +390,6 @@

 unsigned long __ipipe_mach_get_dec(void)
 {
-    return __raw_readl(S3C2410_TCNTO(4));
+    return (unsigned long)__raw_readw(S3C2410_TCNTO(4));
 }
 #endif /* CONFIG_IPIPE */

but it didn't work. (In fact, TCNTOs of S4C2440 are 16-bit, not 32-bit, and
the datasheet does not guarantee that higher bits are read as zeroes; in
fact they are, but I tried explicit conversion anyway just in case. No
effect.).

--
Alex

[-- Attachment #2: Type: text/html, Size: 1802 bytes --]

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Gilles Chanteperdrix]

Alexey Galakhov wrote:
> 2011/5/1 Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> 
>> Ok. Could you try only keeping the part with ipipe_handle_chained_irq,
>> leaving the handle_edge_irqs?
>>
> 
> Sure. I'll do it tomorrow.
> 
> There is another one issue: the board hangs after about 2-3 minutes running.
> I found that it is caused by a semi-permanent (order of 2^64 iterations)
> loop in (untouched) kernel timer code, most likely due to weird return value
> of __ipipe_tsc_get. I haven't found the source of this problem yet. I tried
> a small fix:

So, there is an issue when the asm code wraps. I will try and reproduce
this.

-- 
                                                                Gilles.

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Gilles Chanteperdrix]

Alexey Galakhov wrote:
> 2011/5/1 Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> 
>> Ok. Could you try only keeping the part with ipipe_handle_chained_irq,
>> leaving the handle_edge_irqs?
>>
> 
> Sure. I'll do it tomorrow.
> 
> There is another one issue: the board hangs after about 2-3 minutes running.
> I found that it is caused by a semi-permanent (order of 2^64 iterations)
> loop in (untouched) kernel timer code, most likely due to weird return value
> of __ipipe_tsc_get. I haven't found the source of this problem yet. I tried
> a small fix:
> 
> --- time.c.org    2011-04-26 20:52:00.000000000 +0600
> +++ time.c    2011-05-01 02:03:33.000000000 +0600
> @@ -124,7 +124,7 @@
> 
>  static inline unsigned long timer_freerunning_getvalue(void)
>  {
> -    return __raw_readl(S3C2410_TCNTO(3));
> +    return (unsigned long)__raw_readw(S3C2410_TCNTO(3));
>  }
> 
>  static inline unsigned long timer_freerunning_getticksoffset(unsigned long
> tval)
> @@ -390,6 +390,6 @@
> 
>  unsigned long __ipipe_mach_get_dec(void)
>  {
> -    return __raw_readl(S3C2410_TCNTO(4));
> +    return (unsigned long)__raw_readw(S3C2410_TCNTO(4));
>  }
>  #endif /* CONFIG_IPIPE */
> 
> but it didn't work. (In fact, TCNTOs of S4C2440 are 16-bit, not 32-bit, and
> the datasheet does not guarantee that higher bits are read as zeroes; in
> fact they are, but I tried explicit conversion anyway just in case. No
> effect.).

There is a bug in ipipe_tsc_update for decrementers. The all-in-one patch I 
would like you to try is:

diff --git a/arch/arm/kernel/ipipe_tsc.c b/arch/arm/kernel/ipipe_tsc.c
index a9de4f9..c5c2de0 100644
--- a/arch/arm/kernel/ipipe_tsc.c
+++ b/arch/arm/kernel/ipipe_tsc.c
@@ -104,8 +104,8 @@ void __ipipe_tsc_update(void)
 		int offset = ipipe_tsc_value->last_cnt - cnt;
 		if (offset < 0)
 			offset += 0x10000;
-		ipipe_tsc_value->last_tsc += offset + 1;
-		ipipe_tsc_value->last_cnt = cnt - 1;
+		ipipe_tsc_value->last_tsc += offset;
+		ipipe_tsc_value->last_cnt = cnt;
 		return;
 	}
 	ipipe_tsc_value->last_tsc = __ipipe_tsc_get() - 1;
diff --git a/arch/arm/kernel/ipipe_tsc_asm.S b/arch/arm/kernel/ipipe_tsc_asm.S
index ca88882..d3c833f 100644
--- a/arch/arm/kernel/ipipe_tsc_asm.S
+++ b/arch/arm/kernel/ipipe_tsc_asm.S
@@ -143,7 +143,7 @@ __ipipe_decrementer_16:
 	ldr	ip, [r0]
 	ldr	r2, .LCdec16_last_cnt
 	subs 	ip, r2, ip
-	addcs	ip, ip, #0x10000
+	addcc	ip, ip, #0x10000
 	myldrd	r2, r3, r3, .LCdec16_last_tsc
 	cmp	r1, r2
 	bne	1b
@@ -155,7 +155,7 @@ __ipipe_decrementer_16:
 	ldr	ip, [r0]
 	ldr	r2, .LCdec16_last_cnt
 	subs 	ip, r2, ip
-	addcs	ip, ip, #0x10000
+	addcc	ip, ip, #0x10000
 	myldrd	r2, r3, r3, .LCdec16_last_tsc
 	cmp	r1, r3
 	bne	1b
diff --git a/arch/arm/plat-samsung/irq-uart.c b/arch/arm/plat-samsung/irq-uart.c
index 4f8c102..1da5aff 100644
--- a/arch/arm/plat-samsung/irq-uart.c
+++ b/arch/arm/plat-samsung/irq-uart.c
@@ -88,13 +88,13 @@ static void s3c_irq_demux_uart(unsigned int irq, struct irq_desc *desc)
 	int base = uirq->base_irq;
 
 	if (pend & (1 << 0))
-		generic_handle_irq(base);
+		ipipe_handle_chained_irq(base);
 	if (pend & (1 << 1))
-		generic_handle_irq(base + 1);
+		ipipe_handle_chained_irq(base + 1);
 	if (pend & (1 << 2))
-		generic_handle_irq(base + 2);
+		ipipe_handle_chained_irq(base + 2);
 	if (pend & (1 << 3))
-		generic_handle_irq(base + 3);
+		ipipe_handle_chained_irq(base + 3);
 }
 
 static struct irq_chip s3c_irq_uart = {
diff --git a/arch/arm/plat-samsung/time.c b/arch/arm/plat-samsung/time.c
index 3e9cef2..baf61f6 100644
--- a/arch/arm/plat-samsung/time.c
+++ b/arch/arm/plat-samsung/time.c
@@ -154,7 +154,7 @@ static inline unsigned long getticksoffset_tscupdate(void)
 	last_free_running_tcnt = tval;
 	__ipipe_tsc_update();
 	return ticks;
-	}
+}
 #else
 static unsigned long s3c2410_gettimeoffset (void)
 {


-- 
                                                                Gilles.

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Alexey Galakhov]

[-- Attachment #1: Type: text/plain, Size: 512 bytes --]

2011/5/1 Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>

>
> There is a bug in ipipe_tsc_update for decrementers. The all-in-one patch I
> would like you to try is:
>
>
This hangs ("infinite IRQs 30,37" symptom). It works fine after re-applying
this:
- set_irq_handler(irqno, handle_edge_irq);
+ set_irq_handler(irqno, handle_level_irq);

Seems that handle_edge_irq is broken while handle_level_irq works just fine.

Now the board seems to work. At least, there are no obvious hangups. Thanks!

--
Alex

[-- Attachment #2: Type: text/html, Size: 849 bytes --]

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Gilles Chanteperdrix]

On 05/01/2011 04:23 PM, Alexey Galakhov wrote:
> 2011/5/1 Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> 
>>
>> There is a bug in ipipe_tsc_update for decrementers. The all-in-one patch I
>> would like you to try is:
>>
>>
> This hangs ("infinite IRQs 30,37" symptom). It works fine after re-applying
> this:
> - set_irq_handler(irqno, handle_edge_irq);
> + set_irq_handler(irqno, handle_level_irq);
> 
> Seems that handle_edge_irq is broken while handle_level_irq works just fine.
> 
> Now the board seems to work. At least, there are no obvious hangups. Thanks!

Ok. I am going to merge the three changes. Thanks for your patience.

-- 
                                                                Gilles.

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Alexey Galakhov]

[-- Attachment #1: Type: text/plain, Size: 231 bytes --]

2011/4/28 Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>

>
>
> Well, is not that an effect of kgdb?
>

I turned off kgdb completely. My gdb uses openocd and jtag cable, so the
performance should not be affected.

--
Alex

[-- Attachment #2: Type: text/html, Size: 568 bytes --]

Re: [Adeos-main] Ipipe hangs on ARM MINI2440 after switching clocksource

[Gilles Chanteperdrix]

Gilles Chanteperdrix wrote:
> Alexey Galakhov wrote:
>> On 04/27/2011 09:56 PM, Gilles Chanteperdrix wrote:
>>> But I think your problem has more to do with the following commit, could
>>> you try reverting it?
>>> http://git.xenomai.org/?p=ipipe-gch.git;a=commitdiff;h=6ab29d9c7a4b119f45ef4d93780e894fe1c0c6c6;hp=cd9c5e016092258d4450e137be2d0844d0fe8b38
>>>
>>> The same issue was reported on ixp4xx.
>> Just tried to revert this patch. Hangs. But now in a different way: it
>> starts to initialize peripherals and stops on LED initialization. Looks
>> like race conditions since it stops in slightly different place each
>> time I try to boot (randomly at led3, led4 or "backlight" LED). Last
>> lines of dmesg are, i.e.:
>>
>> [    3.090000] samsung-ts s3c2440-ts: driver attached, registering input
>> device
>> [    3.100000] input: S3C24XX TouchScreen as /devices/virtual/input/input0
>> [    3.120000] S3C24XX RTC, (c) 2004,2006 Simtec Electronics
>> [    3.120000] s3c-rtc s3c2410-rtc: rtc disabled, re-enabling
>> [    3.135000] s3c-rtc s3c2410-rtc: rtc core: registered s3c as rtc0
>> [    3.135000] i2c /dev entries driver
>> [    3.155000] S3C2410 Watchdog Timer, (c) 2004 Simtec Electronics
>> [    3.160000] s3c2410-wdt s3c2410-wdt: watchdog inactive, reset
>> disabled, irq enabled
>> [    3.180000] s3c-sdi s3c2440-sdi: powered down.
>> [    3.180000] s3c-sdi s3c2440-sdi: mmc0 - using pio, sw SDIO IRQ
>> [    3.185000] s3c-sdi s3c2440-sdi: running at 0kHz (requested: 0kHz).
>> [    3.200000] Registered led device: led1
>> [    3.205000] Registered led device: led2
>> [    3.210000] s3c-sdi s3c2440-sdi: running at 398kHz (requested: 400kHz).
>> [    3.215000] Registered led device: led3
>> [    3.220000] Registered led device: led4
>> (hangs)
>>
>> Going to try kgdb over serial line.
> 
> Ok. Better try without this patch first, then try reapplying the patch
> when you are certain that everything else works correctly. The patch
> enables some assembly code that was never actually tested. On my side, I
> will try and test it separately.
> 

With the following patch, the asm code looks fine to me:

diff --git a/arch/arm/kernel/ipipe_tsc_asm.S
b/arch/arm/kernel/ipipe_tsc_asm.S
index ca88882..d3c833f 100644
--- a/arch/arm/kernel/ipipe_tsc_asm.S
+++ b/arch/arm/kernel/ipipe_tsc_asm.S
@@ -143,7 +143,7 @@ __ipipe_decrementer_16:
 	ldr	ip, [r0]
 	ldr	r2, .LCdec16_last_cnt
 	subs 	ip, r2, ip
-	addcs	ip, ip, #0x10000
+	addcc	ip, ip, #0x10000
 	myldrd	r2, r3, r3, .LCdec16_last_tsc
 	cmp	r1, r2
 	bne	1b
@@ -155,7 +155,7 @@ __ipipe_decrementer_16:
 	ldr	ip, [r0]
 	ldr	r2, .LCdec16_last_cnt
 	subs 	ip, r2, ip
-	addcs	ip, ip, #0x10000
+	addcc	ip, ip, #0x10000
 	myldrd	r2, r3, r3, .LCdec16_last_tsc
 	cmp	r1, r3
 	bne	1b



-- 
                                                                Gilles.