OMAP3 ISP deadlocks on my new arm

OMAP3 ISP deadlocks on my new arm

[Bastian Hecht]

Hello people,

I switched to the new DM3730 from IGEP and while it's supposed to be
(almost) the same as the 3530 Version the isp deadlocks
deterministically after I start capturing the second time with yavta.
All extra locking debug output is enabled in the kernel .config.

I am unsure if this is an ISP thing or a problem in the
interrupthandling software.
The first block is the watchdog that detects the deadlock. The last
block is in the isp-code but how can it hang when trying to UNlock a
spinlock? I am unsure about the 2nd block.
The assembler code of __irq_svc is located in arch/arm/kernel/entry-armv.S
Maybe I should try on linux-arm@lists.arm.linux.org.uk but I thought I
give it a shot here first.

I use the omap3isp-2.6.35.3-omap3isp branch from Laurent.

Any ideas? Thanks for any help,

 Bastian Hecht


[  190.059509] BUG: soft lockup - CPU#0 stuck for 61s! [yavta:2224]
[  190.065704] Kernel panic - not syncing: softlockup: hung tasks
[  190.071563] [<c0031078>] (unwind_backtrace+0x0/0xe4) from
[<c02baf24>] (panic+0x50/0xd0)
[  190.079711] [<c02baf24>] (panic+0x50/0xd0) from [<c00729e4>]
(softlockup_tick+0x134/0x158)
[  190.088043] [<c00729e4>] (softlockup_tick+0x134/0x158) from
[<c005612c>] (update_process_times+0x28/0x48)
[  190.097656] [<c005612c>] (update_process_times+0x28/0x48) from
[<c00697bc>] (tick_sched_timer+0x88/0xbc)
[  190.107177] [<c00697bc>] (tick_sched_timer+0x88/0xbc) from
[<c0061ff0>] (__run_hrtimer+0x44/0x84)
[  190.116119] [<c0061ff0>] (__run_hrtimer+0x44/0x84) from
[<c0062144>] (hrtimer_interrupt+0x114/0x2c8)
[  190.125305] [<c0062144>] (hrtimer_interrupt+0x114/0x2c8) from
[<c0035e20>] (omap2_gp_timer_interrupt+0x20/0x2c)
[  190.135437] [<c0035e20>] (omap2_gp_timer_interrupt+0x20/0x2c) from
[<c00730e4>] (handle_IRQ_event+0x24/0xe0)
[  190.145324] [<c00730e4>] (handle_IRQ_event+0x24/0xe0) from
[<c0074b80>] (handle_level_irq+0x90/0xfc)
[  190.154510] [<c0074b80>] (handle_level_irq+0x90/0xfc) from
[<c002c06c>] (asm_do_IRQ+0x6c/0x8c)
[  190.163177] [<c002c06c>] (asm_do_IRQ+0x6c/0x8c) from [<c002c9f4>]
(__irq_svc+0x34/0x80)
[  190.171234] Exception stack(0xda413c98 to 0xda413ce0)
[  190.176300] 3c80:
    00000020 c03c20d0
[  190.184509] 3ca0: 00000000 c0417240 da412000 00000002 00000000
ded72e84 0000000a dec54640
[  190.192718] 3cc0: c0417240 00000000 00000002 da413ce0 c0051bd4
c0051ad8 20000113 ffffffff
[  190.200958] [<c002c9f4>] (__irq_svc+0x34/0x80) from [<c0051ad8>]
(__do_softirq+0x3c/0xf8)
[  190.209167] [<c0051ad8>] (__do_softirq+0x3c/0xf8) from [<c0051bd4>]
(irq_exit+0x40/0x8c)
[  190.217315] [<c0051bd4>] (irq_exit+0x40/0x8c) from [<c002c070>]
(asm_do_IRQ+0x70/0x8c)
[  190.225280] [<c002c070>] (asm_do_IRQ+0x70/0x8c) from [<c002c9f4>]
(__irq_svc+0x34/0x80)
[  190.233306] Exception stack(0xda413d20 to 0xda413d68)
[  190.238403] 3d20: defc4938 defc48c0 defc4084 ded72e84 ded72e14
ded72e80 40000013 ded72e84
[  190.246612] 3d40: ded72e68 dec54640 dedc5a38 00000000 defc40f8
da413d68 bf01d4cc bf01d4ec
[  190.254821] 3d60: 60000013 ffffffff
[  190.258361] [<c002c9f4>] (__irq_svc+0x34/0x80) from [<bf01d4ec>]
(omap3isp_video_queue_streamon+0x6c/0x7c [omap3_isp])
[  190.269165] [<bf01d4ec>] (omap3isp_video_queue_streamon+0x6c/0x7c
[omap3_isp]) from [<bf01f1d4>] (isp_video_streamon+0x150/0x1f8
[omap3_isp])
[  190.281951] [<bf01f1d4>] (isp_video_streamon+0x150/0x1f8
[omap3_isp]) from [<c01fa76c>] (__video_do_ioctl+0x1488/0x3bd0)
[  190.292877] [<c01fa76c>] (__video_do_ioctl+0x1488/0x3bd0) from
[<c01f8ee8>] (__video_usercopy+0x2d0/0x414)
[  190.302581] [<c01f8ee8>] (__video_usercopy+0x2d0/0x414) from
[<c01f8370>] (v4l2_unlocked_ioctl+0x38/0x3c)
[  190.312194] [<c01f8370>] (v4l2_unlocked_ioctl+0x38/0x3c) from
[<c00a6d6c>] (vfs_ioctl+0x2c/0x6c)
[  190.321044] [<c00a6d6c>] (vfs_ioctl+0x2c/0x6c) from [<c00a7448>]
(do_vfs_ioctl+0x4cc/0x514)
[  190.329437] [<c00a7448>] (do_vfs_ioctl+0x4cc/0x514) from
[<c00a74c4>] (sys_ioctl+0x34/0x54)
[  190.337829] [<c00a74c4>] (sys_ioctl+0x34/0x54) from [<c002ce40>]
(ret_fast_syscall+0x0/0x30)

Re: OMAP3 ISP deadlocks on my new arm

[Bastian Hecht]

Hello,

I attached the output without extra kernel lock info, here is the debug output:


[  375.811157] BUG: soft lockup - CPU#0 stuck for 61s! [yavta:2226]
[  375.817474] Kernel panic - not syncing: softlockup: hung tasks
[  375.823364] [<c003250c>] (unwind_backtrace+0x0/0xe4) from
[<c02e11a0>] (panic+0x50/0xd4)
[  375.831512] [<c02e11a0>] (panic+0x50/0xd4) from [<c007e154>]
(softlockup_tick+0x14c/0x170)
[  375.839813] [<c007e154>] (softlockup_tick+0x14c/0x170) from
[<c00592e8>] (update_process_times+0x28/0x48)
[  375.849456] [<c00592e8>] (update_process_times+0x28/0x48) from
[<c006e840>] (tick_sched_timer+0x88/0xbc)
[  375.858978] [<c006e840>] (tick_sched_timer+0x88/0xbc) from
[<c00666c4>] (__run_hrtimer+0x50/0x9c)
[  375.867889] [<c00666c4>] (__run_hrtimer+0x50/0x9c) from
[<c006681c>] (hrtimer_interrupt+0x10c/0x2d8)
[  375.877075] [<c006681c>] (hrtimer_interrupt+0x10c/0x2d8) from
[<c0037438>] (omap2_gp_timer_interrupt+0x20/0x2c)
[  375.887237] [<c0037438>] (omap2_gp_timer_interrupt+0x20/0x2c) from
[<c007e944>] (handle_IRQ_event+0x24/0xe4)
[  375.897125] [<c007e944>] (handle_IRQ_event+0x24/0xe4) from
[<c0080570>] (handle_level_irq+0xac/0x128)
[  375.906402] [<c0080570>] (handle_level_irq+0xac/0x128) from
[<c002d06c>] (asm_do_IRQ+0x6c/0x8c)
[  375.915130] [<c002d06c>] (asm_do_IRQ+0x6c/0x8c) from [<c002da78>]
(__irq_svc+0x38/0xa0)
[  375.923187] Exception stack(0xdea1dc80 to 0xdea1dcc8)
[  375.928253] dc80: 00000001 dea3e840 00000110 0001dbb7 dea1c000
00000002 00000000 dff0cac8
[  375.936492] dca0: 0000000a deab8800 c0461080 00000000 c0773214
dea1dcc8 c0071ba0 c0054614
[  375.944702] dcc0: 60000113 ffffffff
[  375.948211] [<c002da78>] (__irq_svc+0x38/0xa0) from [<c0054614>]
(__do_softirq+0x4c/0x128)
[  375.956512] [<c0054614>] (__do_softirq+0x4c/0x128) from
[<c0054740>] (irq_exit+0x50/0x9c)
[  375.964752] [<c0054740>] (irq_exit+0x50/0x9c) from [<c002d070>]
(asm_do_IRQ+0x70/0x8c)
[  375.972686] [<c002d070>] (asm_do_IRQ+0x70/0x8c) from [<c002da78>]
(__irq_svc+0x38/0xa0)
[  375.980743] Exception stack(0xdea1dd08 to 0xdea1dd50)
[  375.985809] dd00:                   00000001 dea3e840 00000110
0001dbb4 40000013 dff0caa8
[  375.994049] dd20: dff0cac4 dff0cac8 dff0caa8 deab8800 40000013
00000000 c089e5fc dea1dd50
[  376.002258] dd40: c0071ba0 c02e3dc4 20000013 ffffffff
[  376.007354] [<c002da78>] (__irq_svc+0x38/0xa0) from [<c02e3dc4>]
(_raw_spin_unlock_irqrestore+0x40/0x44)
[  376.016906] [<c02e3dc4>] (_raw_spin_unlock_irqrestore+0x40/0x44)
from [<bf01f678>] (omap3isp_video_queue_streamon+0x80/0x90
[omap3_isp])
[  376.029388] [<bf01f678>] (omap3isp_video_queue_streamon+0x80/0x90
[omap3_isp]) from [<bf02128c>] (isp_video_streamon+0x15c/0x214
[omap3_isp])
[  376.042175] [<bf02128c>] (isp_video_streamon+0x15c/0x214
[omap3_isp]) from [<c0216b38>] (__video_do_ioctl+0x1488/0x3bd0)
[  376.053100] [<c0216b38>] (__video_do_ioctl+0x1488/0x3bd0) from
[<c02152b4>] (__video_usercopy+0x2d0/0x414)
[  376.062835] [<c02152b4>] (__video_usercopy+0x2d0/0x414) from
[<c0214708>] (v4l2_unlocked_ioctl+0x38/0x3c)
[  376.072448] [<c0214708>] (v4l2_unlocked_ioctl+0x38/0x3c) from
[<c00b5f1c>] (vfs_ioctl+0x2c/0x6c)
[  376.081268] [<c00b5f1c>] (vfs_ioctl+0x2c/0x6c) from [<c00b6610>]
(do_vfs_ioctl+0x4e4/0x52c)
[  376.089660] [<c00b6610>] (do_vfs_ioctl+0x4e4/0x52c) from
[<c00b668c>] (sys_ioctl+0x34/0x54)
[  376.098052] [<c00b668c>] (sys_ioctl+0x34/0x54) from [<c002df40>]
(ret_fast_syscall+0x0/0x3c)
[  376.106933] ------------[ cut here ]------------
[  376.111572] WARNING: at kernel/lockdep.c:2327 panic+0xb0/0xd4()
[  376.117523] Modules linked in: board_bastix framix omap3_isp iovmm
omap_iommu iommu2 iommu
[  376.125915] [<c003250c>] (unwind_backtrace+0x0/0xe4) from
[<c004f554>] (warn_slowpath_common+0x4c/0x64)
[  376.135375] [<c004f554>] (warn_slowpath_common+0x4c/0x64) from
[<c004f584>] (warn_slowpath_null+0x18/0x1c)
[  376.145080] [<c004f584>] (warn_slowpath_null+0x18/0x1c) from
[<c02e1200>] (panic+0xb0/0xd4)
[  376.153472] [<c02e1200>] (panic+0xb0/0xd4) from [<c007e154>]
(softlockup_tick+0x14c/0x170)
[  376.161804] [<c007e154>] (softlockup_tick+0x14c/0x170) from
[<c00592e8>] (update_process_times+0x28/0x48)
[  376.171417] [<c00592e8>] (update_process_times+0x28/0x48) from
[<c006e840>] (tick_sched_timer+0x88/0xbc)
[  376.180969] [<c006e840>] (tick_sched_timer+0x88/0xbc) from
[<c00666c4>] (__run_hrtimer+0x50/0x9c)
[  376.189880] [<c00666c4>] (__run_hrtimer+0x50/0x9c) from
[<c006681c>] (hrtimer_interrupt+0x10c/0x2d8)
[  376.199066] [<c006681c>] (hrtimer_interrupt+0x10c/0x2d8) from
[<c0037438>] (omap2_gp_timer_interrupt+0x20/0x2c)
[  376.209197] [<c0037438>] (omap2_gp_timer_interrupt+0x20/0x2c) from
[<c007e944>] (handle_IRQ_event+0x24/0xe4)
[  376.219085] [<c007e944>] (handle_IRQ_event+0x24/0xe4) from
[<c0080570>] (handle_level_irq+0xac/0x128)
[  376.228363] [<c0080570>] (handle_level_irq+0xac/0x128) from
[<c002d06c>] (asm_do_IRQ+0x6c/0x8c)
[  376.237121] [<c002d06c>] (asm_do_IRQ+0x6c/0x8c) from [<c002da78>]
(__irq_svc+0x38/0xa0)
[  376.245147] Exception stack(0xdea1dc80 to 0xdea1dcc8)
[  376.250244] dc80: 00000001 dea3e840 00000110 0001dbb7 dea1c000
00000002 00000000 dff0cac8
[  376.258453] dca0: 0000000a deab8800 c0461080 00000000 c0773214
dea1dcc8 c0071ba0 c0054614
[  376.266662] dcc0: 60000113 ffffffff
[  376.270172] [<c002da78>] (__irq_svc+0x38/0xa0) from [<c0054614>]
(__do_softirq+0x4c/0x128)
[  376.278503] [<c0054614>] (__do_softirq+0x4c/0x128) from
[<c0054740>] (irq_exit+0x50/0x9c)
[  376.286712] [<c0054740>] (irq_exit+0x50/0x9c) from [<c002d070>]
(asm_do_IRQ+0x70/0x8c)
[  376.294677] [<c002d070>] (asm_do_IRQ+0x70/0x8c) from [<c002da78>]
(__irq_svc+0x38/0xa0)
[  376.302703] Exception stack(0xdea1dd08 to 0xdea1dd50)
[  376.307800] dd00:                   00000001 dea3e840 00000110
0001dbb4 40000013 dff0caa8
[  376.316009] dd20: dff0cac4 dff0cac8 dff0caa8 deab8800 40000013
00000000 c089e5fc dea1dd50
[  376.324218] dd40: c0071ba0 c02e3dc4 20000013 ffffffff
[  376.329315] [<c002da78>] (__irq_svc+0x38/0xa0) from [<c02e3dc4>]
(_raw_spin_unlock_irqrestore+0x40/0x44)
[  376.338897] [<c02e3dc4>] (_raw_spin_unlock_irqrestore+0x40/0x44)
from [<bf01f678>] (omap3isp_video_queue_streamon+0x80/0x90
[omap3_isp])
[  376.351257] [<bf01f678>] (omap3isp_video_queue_streamon+0x80/0x90
[omap3_isp]) from [<bf02128c>] (isp_video_streamon+0x15c/0x214
[omap3_isp])
[  376.364044] [<bf02128c>] (isp_video_streamon+0x15c/0x214
[omap3_isp]) from [<c0216b38>] (__video_do_ioctl+0x1488/0x3bd0)
[  376.375000] [<c0216b38>] (__video_do_ioctl+0x1488/0x3bd0) from
[<c02152b4>] (__video_usercopy+0x2d0/0x414)
[  376.384704] [<c02152b4>] (__video_usercopy+0x2d0/0x414) from
[<c0214708>] (v4l2_unlocked_ioctl+0x38/0x3c)
[  376.394317] [<c0214708>] (v4l2_unlocked_ioctl+0x38/0x3c) from
[<c00b5f1c>] (vfs_ioctl+0x2c/0x6c)
[  376.403137] [<c00b5f1c>] (vfs_ioctl+0x2c/0x6c) from [<c00b6610>]
(do_vfs_ioctl+0x4e4/0x52c)
[  376.411529] [<c00b6610>] (do_vfs_ioctl+0x4e4/0x52c) from
[<c00b668c>] (sys_ioctl+0x34/0x54)
[  376.419952] [<c00b668c>] (sys_ioctl+0x34/0x54) from [<c002df40>]
(ret_fast_syscall+0x0/0x3c)
[  376.428436] ---[ end trace 1b75b31a2719ed1e ]---


2011/4/13 Bastian Hecht <hechtb@googlemail.com>:
> Hello people,
>
> I switched to the new DM3730 from IGEP and while it's supposed to be
> (almost) the same as the 3530 Version the isp deadlocks
> deterministically after I start capturing the second time with yavta.
> All extra locking debug output is enabled in the kernel .config.
>
> I am unsure if this is an ISP thing or a problem in the
> interrupthandling software.
> The first block is the watchdog that detects the deadlock. The last
> block is in the isp-code but how can it hang when trying to UNlock a
> spinlock? I am unsure about the 2nd block.
> The assembler code of __irq_svc is located in arch/arm/kernel/entry-armv.S
> Maybe I should try on linux-arm@lists.arm.linux.org.uk but I thought I
> give it a shot here first.
>
> I use the omap3isp-2.6.35.3-omap3isp branch from Laurent.
>
> Any ideas? Thanks for any help,
>
>  Bastian Hecht
>
>
> [  190.059509] BUG: soft lockup - CPU#0 stuck for 61s! [yavta:2224]
> [  190.065704] Kernel panic - not syncing: softlockup: hung tasks
> [  190.071563] [<c0031078>] (unwind_backtrace+0x0/0xe4) from
> [<c02baf24>] (panic+0x50/0xd0)
> [  190.079711] [<c02baf24>] (panic+0x50/0xd0) from [<c00729e4>]
> (softlockup_tick+0x134/0x158)
> [  190.088043] [<c00729e4>] (softlockup_tick+0x134/0x158) from
> [<c005612c>] (update_process_times+0x28/0x48)
> [  190.097656] [<c005612c>] (update_process_times+0x28/0x48) from
> [<c00697bc>] (tick_sched_timer+0x88/0xbc)
> [  190.107177] [<c00697bc>] (tick_sched_timer+0x88/0xbc) from
> [<c0061ff0>] (__run_hrtimer+0x44/0x84)
> [  190.116119] [<c0061ff0>] (__run_hrtimer+0x44/0x84) from
> [<c0062144>] (hrtimer_interrupt+0x114/0x2c8)
> [  190.125305] [<c0062144>] (hrtimer_interrupt+0x114/0x2c8) from
> [<c0035e20>] (omap2_gp_timer_interrupt+0x20/0x2c)
> [  190.135437] [<c0035e20>] (omap2_gp_timer_interrupt+0x20/0x2c) from
> [<c00730e4>] (handle_IRQ_event+0x24/0xe0)
> [  190.145324] [<c00730e4>] (handle_IRQ_event+0x24/0xe0) from
> [<c0074b80>] (handle_level_irq+0x90/0xfc)
> [  190.154510] [<c0074b80>] (handle_level_irq+0x90/0xfc) from
> [<c002c06c>] (asm_do_IRQ+0x6c/0x8c)
> [  190.163177] [<c002c06c>] (asm_do_IRQ+0x6c/0x8c) from [<c002c9f4>]
> (__irq_svc+0x34/0x80)
> [  190.171234] Exception stack(0xda413c98 to 0xda413ce0)
> [  190.176300] 3c80:
>    00000020 c03c20d0
> [  190.184509] 3ca0: 00000000 c0417240 da412000 00000002 00000000
> ded72e84 0000000a dec54640
> [  190.192718] 3cc0: c0417240 00000000 00000002 da413ce0 c0051bd4
> c0051ad8 20000113 ffffffff
> [  190.200958] [<c002c9f4>] (__irq_svc+0x34/0x80) from [<c0051ad8>]
> (__do_softirq+0x3c/0xf8)
> [  190.209167] [<c0051ad8>] (__do_softirq+0x3c/0xf8) from [<c0051bd4>]
> (irq_exit+0x40/0x8c)
> [  190.217315] [<c0051bd4>] (irq_exit+0x40/0x8c) from [<c002c070>]
> (asm_do_IRQ+0x70/0x8c)
> [  190.225280] [<c002c070>] (asm_do_IRQ+0x70/0x8c) from [<c002c9f4>]
> (__irq_svc+0x34/0x80)
> [  190.233306] Exception stack(0xda413d20 to 0xda413d68)
> [  190.238403] 3d20: defc4938 defc48c0 defc4084 ded72e84 ded72e14
> ded72e80 40000013 ded72e84
> [  190.246612] 3d40: ded72e68 dec54640 dedc5a38 00000000 defc40f8
> da413d68 bf01d4cc bf01d4ec
> [  190.254821] 3d60: 60000013 ffffffff
> [  190.258361] [<c002c9f4>] (__irq_svc+0x34/0x80) from [<bf01d4ec>]
> (omap3isp_video_queue_streamon+0x6c/0x7c [omap3_isp])
> [  190.269165] [<bf01d4ec>] (omap3isp_video_queue_streamon+0x6c/0x7c
> [omap3_isp]) from [<bf01f1d4>] (isp_video_streamon+0x150/0x1f8
> [omap3_isp])
> [  190.281951] [<bf01f1d4>] (isp_video_streamon+0x150/0x1f8
> [omap3_isp]) from [<c01fa76c>] (__video_do_ioctl+0x1488/0x3bd0)
> [  190.292877] [<c01fa76c>] (__video_do_ioctl+0x1488/0x3bd0) from
> [<c01f8ee8>] (__video_usercopy+0x2d0/0x414)
> [  190.302581] [<c01f8ee8>] (__video_usercopy+0x2d0/0x414) from
> [<c01f8370>] (v4l2_unlocked_ioctl+0x38/0x3c)
> [  190.312194] [<c01f8370>] (v4l2_unlocked_ioctl+0x38/0x3c) from
> [<c00a6d6c>] (vfs_ioctl+0x2c/0x6c)
> [  190.321044] [<c00a6d6c>] (vfs_ioctl+0x2c/0x6c) from [<c00a7448>]
> (do_vfs_ioctl+0x4cc/0x514)
> [  190.329437] [<c00a7448>] (do_vfs_ioctl+0x4cc/0x514) from
> [<c00a74c4>] (sys_ioctl+0x34/0x54)
> [  190.337829] [<c00a74c4>] (sys_ioctl+0x34/0x54) from [<c002ce40>]
> (ret_fast_syscall+0x0/0x30)
>

Re: OMAP3 ISP deadlocks on my new arm

[Sakari Ailus]

Bastian Hecht wrote:
> Hello people,

Hi Bastian,

I'm cc'ing Laurent.

> I switched to the new DM3730 from IGEP and while it's supposed to be
> (almost) the same as the 3530 Version the isp deadlocks
> deterministically after I start capturing the second time with yavta.

Does the capture work the first time w/o issues?

> All extra locking debug output is enabled in the kernel .config.

I'm not fully certain on what this exactly is that you have below, but
it looks like your system is staying in interrupt context all the time.
My guess is that the ISP is producing interrupts that the driver is not
handling properly, causing the interrupt handler to be called again
immediately.

Do you have the same sensor working on OMAP 3530?

> I am unsure if this is an ISP thing or a problem in the
> interrupthandling software.

This has probably something to do with the ISP driver. :-)

> The first block is the watchdog that detects the deadlock. The last
> block is in the isp-code but how can it hang when trying to UNlock a
> spinlock? I am unsure about the 2nd block.
> The assembler code of __irq_svc is located in arch/arm/kernel/entry-armv.S
> Maybe I should try on linux-arm@lists.arm.linux.org.uk but I thought I
> give it a shot here first.
> 
> I use the omap3isp-2.6.35.3-omap3isp branch from Laurent.

Why so old kernel?

I think you'd be best off using this one:

<URL:http://git.linuxtv.org/pinchartl/media.git?a=shortlog;h=refs/heads/omap3isp-next-omap3isp>

> Any ideas? Thanks for any help,
> 
>  Bastian Hecht
> 
> 
> [  190.059509] BUG: soft lockup - CPU#0 stuck for 61s! [yavta:2224]
> [  190.065704] Kernel panic - not syncing: softlockup: hung tasks
> [  190.071563] [<c0031078>] (unwind_backtrace+0x0/0xe4) from
> [<c02baf24>] (panic+0x50/0xd0)
> [  190.079711] [<c02baf24>] (panic+0x50/0xd0) from [<c00729e4>]
> (softlockup_tick+0x134/0x158)
> [  190.088043] [<c00729e4>] (softlockup_tick+0x134/0x158) from
> [<c005612c>] (update_process_times+0x28/0x48)
> [  190.097656] [<c005612c>] (update_process_times+0x28/0x48) from
> [<c00697bc>] (tick_sched_timer+0x88/0xbc)
> [  190.107177] [<c00697bc>] (tick_sched_timer+0x88/0xbc) from
> [<c0061ff0>] (__run_hrtimer+0x44/0x84)
> [  190.116119] [<c0061ff0>] (__run_hrtimer+0x44/0x84) from
> [<c0062144>] (hrtimer_interrupt+0x114/0x2c8)
> [  190.125305] [<c0062144>] (hrtimer_interrupt+0x114/0x2c8) from
> [<c0035e20>] (omap2_gp_timer_interrupt+0x20/0x2c)
> [  190.135437] [<c0035e20>] (omap2_gp_timer_interrupt+0x20/0x2c) from
> [<c00730e4>] (handle_IRQ_event+0x24/0xe0)
> [  190.145324] [<c00730e4>] (handle_IRQ_event+0x24/0xe0) from
> [<c0074b80>] (handle_level_irq+0x90/0xfc)
> [  190.154510] [<c0074b80>] (handle_level_irq+0x90/0xfc) from
> [<c002c06c>] (asm_do_IRQ+0x6c/0x8c)
> [  190.163177] [<c002c06c>] (asm_do_IRQ+0x6c/0x8c) from [<c002c9f4>]
> (__irq_svc+0x34/0x80)
> [  190.171234] Exception stack(0xda413c98 to 0xda413ce0)
> [  190.176300] 3c80:
>     00000020 c03c20d0
> [  190.184509] 3ca0: 00000000 c0417240 da412000 00000002 00000000
> ded72e84 0000000a dec54640
> [  190.192718] 3cc0: c0417240 00000000 00000002 da413ce0 c0051bd4
> c0051ad8 20000113 ffffffff
> [  190.200958] [<c002c9f4>] (__irq_svc+0x34/0x80) from [<c0051ad8>]
> (__do_softirq+0x3c/0xf8)
> [  190.209167] [<c0051ad8>] (__do_softirq+0x3c/0xf8) from [<c0051bd4>]
> (irq_exit+0x40/0x8c)
> [  190.217315] [<c0051bd4>] (irq_exit+0x40/0x8c) from [<c002c070>]
> (asm_do_IRQ+0x70/0x8c)
> [  190.225280] [<c002c070>] (asm_do_IRQ+0x70/0x8c) from [<c002c9f4>]
> (__irq_svc+0x34/0x80)
> [  190.233306] Exception stack(0xda413d20 to 0xda413d68)
> [  190.238403] 3d20: defc4938 defc48c0 defc4084 ded72e84 ded72e14
> ded72e80 40000013 ded72e84
> [  190.246612] 3d40: ded72e68 dec54640 dedc5a38 00000000 defc40f8
> da413d68 bf01d4cc bf01d4ec
> [  190.254821] 3d60: 60000013 ffffffff
> [  190.258361] [<c002c9f4>] (__irq_svc+0x34/0x80) from [<bf01d4ec>]
> (omap3isp_video_queue_streamon+0x6c/0x7c [omap3_isp])
> [  190.269165] [<bf01d4ec>] (omap3isp_video_queue_streamon+0x6c/0x7c
> [omap3_isp]) from [<bf01f1d4>] (isp_video_streamon+0x150/0x1f8
> [omap3_isp])
> [  190.281951] [<bf01f1d4>] (isp_video_streamon+0x150/0x1f8
> [omap3_isp]) from [<c01fa76c>] (__video_do_ioctl+0x1488/0x3bd0)
> [  190.292877] [<c01fa76c>] (__video_do_ioctl+0x1488/0x3bd0) from
> [<c01f8ee8>] (__video_usercopy+0x2d0/0x414)
> [  190.302581] [<c01f8ee8>] (__video_usercopy+0x2d0/0x414) from
> [<c01f8370>] (v4l2_unlocked_ioctl+0x38/0x3c)
> [  190.312194] [<c01f8370>] (v4l2_unlocked_ioctl+0x38/0x3c) from
> [<c00a6d6c>] (vfs_ioctl+0x2c/0x6c)
> [  190.321044] [<c00a6d6c>] (vfs_ioctl+0x2c/0x6c) from [<c00a7448>]
> (do_vfs_ioctl+0x4cc/0x514)
> [  190.329437] [<c00a7448>] (do_vfs_ioctl+0x4cc/0x514) from
> [<c00a74c4>] (sys_ioctl+0x34/0x54)
> [  190.337829] [<c00a74c4>] (sys_ioctl+0x34/0x54) from [<c002ce40>]
> (ret_fast_syscall+0x0/0x30)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Regards,

-- 
Sakari Ailus
sakari.ailus@maxwell.research.nokia.com

Re: OMAP3 ISP deadlocks on my new arm

[Bastian Hecht]

Hello Sakari,

2011/4/13 Sakari Ailus <sakari.ailus@maxwell.research.nokia.com>:
> Bastian Hecht wrote:
>> Hello people,
>
> Hi Bastian,
>
> I'm cc'ing Laurent.
>
>> I switched to the new DM3730 from IGEP and while it's supposed to be
>> (almost) the same as the 3530 Version the isp deadlocks
>> deterministically after I start capturing the second time with yavta.
>
> Does the capture work the first time w/o issues?

Yes, I can always run yavta once capturing 4 frames (3 skipped, last saved).
It usually deadlocks when running yavta the second time but I had 1
successful 2nd try (out of 20 maybe).


>> All extra locking debug output is enabled in the kernel .config.
>
> I'm not fully certain on what this exactly is that you have below, but
> it looks like your system is staying in interrupt context all the time.
> My guess is that the ISP is producing interrupts that the driver is not
> handling properly, causing the interrupt handler to be called again
> immediately.

Nice! OK, I'd like to fully understand the panic output, maybe you can
help there:
After
[  376.016906] [<c02e3dc4>] (_raw_spin_unlock_irqrestore+0x40/0x44)
from [<bf01f678>] (omap3isp_video_queue_streamon+0x80/0x90
the IRQs get enabled again. Immediately our offending irq wants to get
served but noone is clearing it.
At some time, the timer interrupt triggers the watchdog for a kernel panic.
So the last exception block is the process context that is currently active.
But why are there 2 irq routines displayed? Is the middle one the
hardware handling that causes a software irq to be triggered (upper
block)?

So my next step could be to find the unhandled irq number?

> Do you have the same sensor working on OMAP 3530?

I never had this problem on an OMAP 3530, although I better test it
again with the current setup. I try to get my hands on an 3530 today.

>> I am unsure if this is an ISP thing or a problem in the
>> interrupthandling software.
>
> This has probably something to do with the ISP driver. :-)
>
>> The first block is the watchdog that detects the deadlock. The last
>> block is in the isp-code but how can it hang when trying to UNlock a
>> spinlock? I am unsure about the 2nd block.
>> The assembler code of __irq_svc is located in arch/arm/kernel/entry-armv.S
>> Maybe I should try on linux-arm@lists.arm.linux.org.uk but I thought I
>> give it a shot here first.
>>
>> I use the omap3isp-2.6.35.3-omap3isp branch from Laurent.
>
> Why so old kernel?

I tried a newer version, but there I couldn't get it booting with my
igep. The kernel unpacked and tried to boot but it froze.
I decided to use a version that is officially is supported by the igep team.

Thanks so much for this valuable guess!

Bastian Hecht



> I think you'd be best off using this one:
>
> <URL:http://git.linuxtv.org/pinchartl/media.git?a=shortlog;h=refs/heads/omap3isp-next-omap3isp>
>
>> Any ideas? Thanks for any help,
>>
>>  Bastian Hecht
>>
>>
>> [  190.059509] BUG: soft lockup - CPU#0 stuck for 61s! [yavta:2224]
>> [  190.065704] Kernel panic - not syncing: softlockup: hung tasks
>> [  190.071563] [<c0031078>] (unwind_backtrace+0x0/0xe4) from
>> [<c02baf24>] (panic+0x50/0xd0)
>> [  190.079711] [<c02baf24>] (panic+0x50/0xd0) from [<c00729e4>]
>> (softlockup_tick+0x134/0x158)
>> [  190.088043] [<c00729e4>] (softlockup_tick+0x134/0x158) from
>> [<c005612c>] (update_process_times+0x28/0x48)
>> [  190.097656] [<c005612c>] (update_process_times+0x28/0x48) from
>> [<c00697bc>] (tick_sched_timer+0x88/0xbc)
>> [  190.107177] [<c00697bc>] (tick_sched_timer+0x88/0xbc) from
>> [<c0061ff0>] (__run_hrtimer+0x44/0x84)
>> [  190.116119] [<c0061ff0>] (__run_hrtimer+0x44/0x84) from
>> [<c0062144>] (hrtimer_interrupt+0x114/0x2c8)
>> [  190.125305] [<c0062144>] (hrtimer_interrupt+0x114/0x2c8) from
>> [<c0035e20>] (omap2_gp_timer_interrupt+0x20/0x2c)
>> [  190.135437] [<c0035e20>] (omap2_gp_timer_interrupt+0x20/0x2c) from
>> [<c00730e4>] (handle_IRQ_event+0x24/0xe0)
>> [  190.145324] [<c00730e4>] (handle_IRQ_event+0x24/0xe0) from
>> [<c0074b80>] (handle_level_irq+0x90/0xfc)
>> [  190.154510] [<c0074b80>] (handle_level_irq+0x90/0xfc) from
>> [<c002c06c>] (asm_do_IRQ+0x6c/0x8c)
>> [  190.163177] [<c002c06c>] (asm_do_IRQ+0x6c/0x8c) from [<c002c9f4>]
>> (__irq_svc+0x34/0x80)
>> [  190.171234] Exception stack(0xda413c98 to 0xda413ce0)
>> [  190.176300] 3c80:
>>     00000020 c03c20d0
>> [  190.184509] 3ca0: 00000000 c0417240 da412000 00000002 00000000
>> ded72e84 0000000a dec54640
>> [  190.192718] 3cc0: c0417240 00000000 00000002 da413ce0 c0051bd4
>> c0051ad8 20000113 ffffffff
>> [  190.200958] [<c002c9f4>] (__irq_svc+0x34/0x80) from [<c0051ad8>]
>> (__do_softirq+0x3c/0xf8)
>> [  190.209167] [<c0051ad8>] (__do_softirq+0x3c/0xf8) from [<c0051bd4>]
>> (irq_exit+0x40/0x8c)
>> [  190.217315] [<c0051bd4>] (irq_exit+0x40/0x8c) from [<c002c070>]
>> (asm_do_IRQ+0x70/0x8c)
>> [  190.225280] [<c002c070>] (asm_do_IRQ+0x70/0x8c) from [<c002c9f4>]
>> (__irq_svc+0x34/0x80)
>> [  190.233306] Exception stack(0xda413d20 to 0xda413d68)
>> [  190.238403] 3d20: defc4938 defc48c0 defc4084 ded72e84 ded72e14
>> ded72e80 40000013 ded72e84
>> [  190.246612] 3d40: ded72e68 dec54640 dedc5a38 00000000 defc40f8
>> da413d68 bf01d4cc bf01d4ec
>> [  190.254821] 3d60: 60000013 ffffffff
>> [  190.258361] [<c002c9f4>] (__irq_svc+0x34/0x80) from [<bf01d4ec>]
>> (omap3isp_video_queue_streamon+0x6c/0x7c [omap3_isp])
>> [  190.269165] [<bf01d4ec>] (omap3isp_video_queue_streamon+0x6c/0x7c
>> [omap3_isp]) from [<bf01f1d4>] (isp_video_streamon+0x150/0x1f8
>> [omap3_isp])
>> [  190.281951] [<bf01f1d4>] (isp_video_streamon+0x150/0x1f8
>> [omap3_isp]) from [<c01fa76c>] (__video_do_ioctl+0x1488/0x3bd0)
>> [  190.292877] [<c01fa76c>] (__video_do_ioctl+0x1488/0x3bd0) from
>> [<c01f8ee8>] (__video_usercopy+0x2d0/0x414)
>> [  190.302581] [<c01f8ee8>] (__video_usercopy+0x2d0/0x414) from
>> [<c01f8370>] (v4l2_unlocked_ioctl+0x38/0x3c)
>> [  190.312194] [<c01f8370>] (v4l2_unlocked_ioctl+0x38/0x3c) from
>> [<c00a6d6c>] (vfs_ioctl+0x2c/0x6c)
>> [  190.321044] [<c00a6d6c>] (vfs_ioctl+0x2c/0x6c) from [<c00a7448>]
>> (do_vfs_ioctl+0x4cc/0x514)
>> [  190.329437] [<c00a7448>] (do_vfs_ioctl+0x4cc/0x514) from
>> [<c00a74c4>] (sys_ioctl+0x34/0x54)
>> [  190.337829] [<c00a74c4>] (sys_ioctl+0x34/0x54) from [<c002ce40>]
>> (ret_fast_syscall+0x0/0x30)
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-media" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
> Regards,
>
> --
> Sakari Ailus
> sakari.ailus@maxwell.research.nokia.com
>

Re: OMAP3 ISP deadlocks on my new arm

[Laurent Pinchart]

Hi Bastian,

On Thursday 14 April 2011 10:33:12 Bastian Hecht wrote:
> 2011/4/13 Sakari Ailus <sakari.ailus@maxwell.research.nokia.com>:
> > Bastian Hecht wrote:
> >> Hello people,
> > 
> > Hi Bastian,
> > 
> > I'm cc'ing Laurent.
> > 
> >> I switched to the new DM3730 from IGEP and while it's supposed to be
> >> (almost) the same as the 3530 Version the isp deadlocks
> >> deterministically after I start capturing the second time with yavta.
> > 
> > Does the capture work the first time w/o issues?
> 
> Yes, I can always run yavta once capturing 4 frames (3 skipped, last
> saved). It usually deadlocks when running yavta the second time but I had
> 1 successful 2nd try (out of 20 maybe).
> 
> >> All extra locking debug output is enabled in the kernel .config.
> > 
> > I'm not fully certain on what this exactly is that you have below, but
> > it looks like your system is staying in interrupt context all the time.
> > My guess is that the ISP is producing interrupts that the driver is not
> > handling properly, causing the interrupt handler to be called again
> > immediately.
> 
> Nice! OK, I'd like to fully understand the panic output, maybe you can
> help there:
> After
> [  376.016906] [<c02e3dc4>] (_raw_spin_unlock_irqrestore+0x40/0x44)
> from [<bf01f678>] (omap3isp_video_queue_streamon+0x80/0x90
> the IRQs get enabled again. Immediately our offending irq wants to get
> served but noone is clearing it.
> At some time, the timer interrupt triggers the watchdog for a kernel panic.
> So the last exception block is the process context that is currently
> active. But why are there 2 irq routines displayed? Is the middle one the
> hardware handling that causes a software irq to be triggered (upper
> block)?
> 
> So my next step could be to find the unhandled irq number?

If the problem is caused by an interrupt storm, the following patch will make 
your system responsive again after a couple of seconds (but will kill the ISP 
driver :-)). If it doesn't, the problem is likely caused by something else.

diff --git a/drivers/media/video/omap3isp/isp.c 
b/drivers/media/video/omap3isp/isp.c
index de2dec5..6497300 100644
--- a/drivers/media/video/omap3isp/isp.c
+++ b/drivers/media/video/omap3isp/isp.c
@@ -462,6 +464,7 @@ static irqreturn_t isp_isr(int irq, void *_isp)
 				       IRQ0STATUS_CCDC_VD0_IRQ |
 				       IRQ0STATUS_CCDC_VD1_IRQ |
 				       IRQ0STATUS_HS_VS_IRQ;
+	static unsigned int count = 0;
 	struct isp_device *isp = _isp;
 	u32 irqstatus;
 	int ret;
@@ -469,6 +472,11 @@ static irqreturn_t isp_isr(int irq, void *_isp)
 	irqstatus = isp_reg_readl(isp, OMAP3_ISP_IOMEM_MAIN, ISP_IRQ0STATUS);
 	isp_reg_writel(isp, irqstatus, OMAP3_ISP_IOMEM_MAIN, ISP_IRQ0STATUS);
 
+	if (count++ > 100000) {
+		isp_disable_interrupts(isp);
+		return IRQ_HANDLED;
+	}
+
 	isp_isr_sbl(isp);
 
 	if (irqstatus & IRQ0STATUS_CSIA_IRQ) {


> > Do you have the same sensor working on OMAP 3530?
> 
> I never had this problem on an OMAP 3530, although I better test it
> again with the current setup. I try to get my hands on an 3530 today.
> 
> >> I am unsure if this is an ISP thing or a problem in the
> >> interrupthandling software.
> > 
> > This has probably something to do with the ISP driver. :-)
> > 
> >> The first block is the watchdog that detects the deadlock. The last
> >> block is in the isp-code but how can it hang when trying to UNlock a
> >> spinlock? I am unsure about the 2nd block.
> >> The assembler code of __irq_svc is located in
> >> arch/arm/kernel/entry-armv.S Maybe I should try on
> >> linux-arm@lists.arm.linux.org.uk but I thought I give it a shot here
> >> first.
> >> 
> >> I use the omap3isp-2.6.35.3-omap3isp branch from Laurent.
> > 
> > Why so old kernel?
> 
> I tried a newer version, but there I couldn't get it booting with my
> igep. The kernel unpacked and tried to boot but it froze.
> I decided to use a version that is officially is supported by the igep
> team.

The ttyS* OMAP serial devices have been renamed to ttyO* in 2.6.37. Replace 
/dev/ttyS2 with /dev/ttyO2 on your kernel command line (either in the kernel 
config file if you've activated CONFIG_CMDLINE_FORCE, or in the boot loader 
environment variables) and you should be able to boot 2.6.38 without any 
issue. Don't forget to modify /etc/inittab as well.

-- 
Regards,

Laurent Pinchart

Re: OMAP3 ISP deadlocks on my new arm

[Bastian Hecht]

Yeah!

Soooo... when I initialized the the camera (loading a 108 bytes
register listing) I just let run the camera and sent images.  So I
first realized a counter overflow  if (count++ > 100000) after a few
seconds. But this seemed to be handled correctly (ignore and delete
HS_VS_IRQ flag) while after 2 or more yavta calls it made the driver
hang.

I modified my register listing so that the chip gets "booted" silently.
In
static struct v4l2_subdev_video_ops framix_subdev_video_ops = {
        .s_stream       = framix_s_stream, <===============
};
I correctly check the stream status now and enable/disable the camera signals.

I am unsure whether the isp should be able to handle an ongoing data
stream independently of ISP_PIPELINE_STREAM_STOPPED.
If you decide it should, I will gladly help you find out more, just
ask. It worked on an OMAP3530 before, but I do not know if it is the
hardware or isp software that changed. Unfortunately I can't get an
3530 anymore (I trashed mine...).


You helped me so much! Big thanks.

Bastian Hecht



2011/4/14 Laurent Pinchart <laurent.pinchart@ideasonboard.com>:
> Hi Bastian,
>
> On Thursday 14 April 2011 10:33:12 Bastian Hecht wrote:
>> 2011/4/13 Sakari Ailus <sakari.ailus@maxwell.research.nokia.com>:
>> > Bastian Hecht wrote:
>> >> Hello people,
>> >
>> > Hi Bastian,
>> >
>> > I'm cc'ing Laurent.
>> >
>> >> I switched to the new DM3730 from IGEP and while it's supposed to be
>> >> (almost) the same as the 3530 Version the isp deadlocks
>> >> deterministically after I start capturing the second time with yavta.
>> >
>> > Does the capture work the first time w/o issues?
>>
>> Yes, I can always run yavta once capturing 4 frames (3 skipped, last
>> saved). It usually deadlocks when running yavta the second time but I had
>> 1 successful 2nd try (out of 20 maybe).
>>
>> >> All extra locking debug output is enabled in the kernel .config.
>> >
>> > I'm not fully certain on what this exactly is that you have below, but
>> > it looks like your system is staying in interrupt context all the time.
>> > My guess is that the ISP is producing interrupts that the driver is not
>> > handling properly, causing the interrupt handler to be called again
>> > immediately.
>>
>> Nice! OK, I'd like to fully understand the panic output, maybe you can
>> help there:
>> After
>> [  376.016906] [<c02e3dc4>] (_raw_spin_unlock_irqrestore+0x40/0x44)
>> from [<bf01f678>] (omap3isp_video_queue_streamon+0x80/0x90
>> the IRQs get enabled again. Immediately our offending irq wants to get
>> served but noone is clearing it.
>> At some time, the timer interrupt triggers the watchdog for a kernel panic.
>> So the last exception block is the process context that is currently
>> active. But why are there 2 irq routines displayed? Is the middle one the
>> hardware handling that causes a software irq to be triggered (upper
>> block)?
>>
>> So my next step could be to find the unhandled irq number?
>
> If the problem is caused by an interrupt storm, the following patch will make
> your system responsive again after a couple of seconds (but will kill the ISP
> driver :-)). If it doesn't, the problem is likely caused by something else.
>
> diff --git a/drivers/media/video/omap3isp/isp.c
> b/drivers/media/video/omap3isp/isp.c
> index de2dec5..6497300 100644
> --- a/drivers/media/video/omap3isp/isp.c
> +++ b/drivers/media/video/omap3isp/isp.c
> @@ -462,6 +464,7 @@ static irqreturn_t isp_isr(int irq, void *_isp)
>                                       IRQ0STATUS_CCDC_VD0_IRQ |
>                                       IRQ0STATUS_CCDC_VD1_IRQ |
>                                       IRQ0STATUS_HS_VS_IRQ;
> +       static unsigned int count = 0;
>        struct isp_device *isp = _isp;
>        u32 irqstatus;
>        int ret;
> @@ -469,6 +472,11 @@ static irqreturn_t isp_isr(int irq, void *_isp)
>        irqstatus = isp_reg_readl(isp, OMAP3_ISP_IOMEM_MAIN, ISP_IRQ0STATUS);
>        isp_reg_writel(isp, irqstatus, OMAP3_ISP_IOMEM_MAIN, ISP_IRQ0STATUS);
>
> +       if (count++ > 100000) {
> +               isp_disable_interrupts(isp);
> +               return IRQ_HANDLED;
> +       }
> +
>        isp_isr_sbl(isp);
>
>        if (irqstatus & IRQ0STATUS_CSIA_IRQ) {
>
>
>> > Do you have the same sensor working on OMAP 3530?
>>
>> I never had this problem on an OMAP 3530, although I better test it
>> again with the current setup. I try to get my hands on an 3530 today.
>>
>> >> I am unsure if this is an ISP thing or a problem in the
>> >> interrupthandling software.
>> >
>> > This has probably something to do with the ISP driver. :-)
>> >
>> >> The first block is the watchdog that detects the deadlock. The last
>> >> block is in the isp-code but how can it hang when trying to UNlock a
>> >> spinlock? I am unsure about the 2nd block.
>> >> The assembler code of __irq_svc is located in
>> >> arch/arm/kernel/entry-armv.S Maybe I should try on
>> >> linux-arm@lists.arm.linux.org.uk but I thought I give it a shot here
>> >> first.
>> >>
>> >> I use the omap3isp-2.6.35.3-omap3isp branch from Laurent.
>> >
>> > Why so old kernel?
>>
>> I tried a newer version, but there I couldn't get it booting with my
>> igep. The kernel unpacked and tried to boot but it froze.
>> I decided to use a version that is officially is supported by the igep
>> team.
>
> The ttyS* OMAP serial devices have been renamed to ttyO* in 2.6.37. Replace
> /dev/ttyS2 with /dev/ttyO2 on your kernel command line (either in the kernel
> config file if you've activated CONFIG_CMDLINE_FORCE, or in the boot loader
> environment variables) and you should be able to boot 2.6.38 without any
> issue. Don't forget to modify /etc/inittab as well.
>
> --
> Regards,
>
> Laurent Pinchart
>

Re: OMAP3 ISP deadlocks on my new arm

[David Cohen]

Hi Bastian,

On Thu, Apr 14, 2011 at 1:36 PM, Bastian Hecht <hechtb@googlemail.com> wrote:
> Yeah!
>
> Soooo... when I initialized the the camera (loading a 108 bytes
> register listing) I just let run the camera and sent images.  So I
> first realized a counter overflow  if (count++ > 100000) after a few
> seconds. But this seemed to be handled correctly (ignore and delete
> HS_VS_IRQ flag) while after 2 or more yavta calls it made the driver
> hang.
>
> I modified my register listing so that the chip gets "booted" silently.
> In
> static struct v4l2_subdev_video_ops framix_subdev_video_ops = {
>        .s_stream       = framix_s_stream, <===============
> };
> I correctly check the stream status now and enable/disable the camera signals.
>
> I am unsure whether the isp should be able to handle an ongoing data
> stream independently of ISP_PIPELINE_STREAM_STOPPED.

streamoff should finish synchronously with last ongoing data. So, it
should have no ongoing data afterwards. Was that your question?

Br,

David Cohen

> If you decide it should, I will gladly help you find out more, just
> ask. It worked on an OMAP3530 before, but I do not know if it is the
> hardware or isp software that changed. Unfortunately I can't get an
> 3530 anymore (I trashed mine...).
>
>
> You helped me so much! Big thanks.
>
> Bastian Hecht
>
>
>
> 2011/4/14 Laurent Pinchart <laurent.pinchart@ideasonboard.com>:
>> Hi Bastian,
>>
>> On Thursday 14 April 2011 10:33:12 Bastian Hecht wrote:
>>> 2011/4/13 Sakari Ailus <sakari.ailus@maxwell.research.nokia.com>:
>>> > Bastian Hecht wrote:
>>> >> Hello people,
>>> >
>>> > Hi Bastian,
>>> >
>>> > I'm cc'ing Laurent.
>>> >
>>> >> I switched to the new DM3730 from IGEP and while it's supposed to be
>>> >> (almost) the same as the 3530 Version the isp deadlocks
>>> >> deterministically after I start capturing the second time with yavta.
>>> >
>>> > Does the capture work the first time w/o issues?
>>>
>>> Yes, I can always run yavta once capturing 4 frames (3 skipped, last
>>> saved). It usually deadlocks when running yavta the second time but I had
>>> 1 successful 2nd try (out of 20 maybe).
>>>
>>> >> All extra locking debug output is enabled in the kernel .config.
>>> >
>>> > I'm not fully certain on what this exactly is that you have below, but
>>> > it looks like your system is staying in interrupt context all the time.
>>> > My guess is that the ISP is producing interrupts that the driver is not
>>> > handling properly, causing the interrupt handler to be called again
>>> > immediately.
>>>
>>> Nice! OK, I'd like to fully understand the panic output, maybe you can
>>> help there:
>>> After
>>> [  376.016906] [<c02e3dc4>] (_raw_spin_unlock_irqrestore+0x40/0x44)
>>> from [<bf01f678>] (omap3isp_video_queue_streamon+0x80/0x90
>>> the IRQs get enabled again. Immediately our offending irq wants to get
>>> served but noone is clearing it.
>>> At some time, the timer interrupt triggers the watchdog for a kernel panic.
>>> So the last exception block is the process context that is currently
>>> active. But why are there 2 irq routines displayed? Is the middle one the
>>> hardware handling that causes a software irq to be triggered (upper
>>> block)?
>>>
>>> So my next step could be to find the unhandled irq number?
>>
>> If the problem is caused by an interrupt storm, the following patch will make
>> your system responsive again after a couple of seconds (but will kill the ISP
>> driver :-)). If it doesn't, the problem is likely caused by something else.
>>
>> diff --git a/drivers/media/video/omap3isp/isp.c
>> b/drivers/media/video/omap3isp/isp.c
>> index de2dec5..6497300 100644
>> --- a/drivers/media/video/omap3isp/isp.c
>> +++ b/drivers/media/video/omap3isp/isp.c
>> @@ -462,6 +464,7 @@ static irqreturn_t isp_isr(int irq, void *_isp)
>>                                       IRQ0STATUS_CCDC_VD0_IRQ |
>>                                       IRQ0STATUS_CCDC_VD1_IRQ |
>>                                       IRQ0STATUS_HS_VS_IRQ;
>> +       static unsigned int count = 0;
>>        struct isp_device *isp = _isp;
>>        u32 irqstatus;
>>        int ret;
>> @@ -469,6 +472,11 @@ static irqreturn_t isp_isr(int irq, void *_isp)
>>        irqstatus = isp_reg_readl(isp, OMAP3_ISP_IOMEM_MAIN, ISP_IRQ0STATUS);
>>        isp_reg_writel(isp, irqstatus, OMAP3_ISP_IOMEM_MAIN, ISP_IRQ0STATUS);
>>
>> +       if (count++ > 100000) {
>> +               isp_disable_interrupts(isp);
>> +               return IRQ_HANDLED;
>> +       }
>> +
>>        isp_isr_sbl(isp);
>>
>>        if (irqstatus & IRQ0STATUS_CSIA_IRQ) {
>>
>>
>>> > Do you have the same sensor working on OMAP 3530?
>>>
>>> I never had this problem on an OMAP 3530, although I better test it
>>> again with the current setup. I try to get my hands on an 3530 today.
>>>
>>> >> I am unsure if this is an ISP thing or a problem in the
>>> >> interrupthandling software.
>>> >
>>> > This has probably something to do with the ISP driver. :-)
>>> >
>>> >> The first block is the watchdog that detects the deadlock. The last
>>> >> block is in the isp-code but how can it hang when trying to UNlock a
>>> >> spinlock? I am unsure about the 2nd block.
>>> >> The assembler code of __irq_svc is located in
>>> >> arch/arm/kernel/entry-armv.S Maybe I should try on
>>> >> linux-arm@lists.arm.linux.org.uk but I thought I give it a shot here
>>> >> first.
>>> >>
>>> >> I use the omap3isp-2.6.35.3-omap3isp branch from Laurent.
>>> >
>>> > Why so old kernel?
>>>
>>> I tried a newer version, but there I couldn't get it booting with my
>>> igep. The kernel unpacked and tried to boot but it froze.
>>> I decided to use a version that is officially is supported by the igep
>>> team.
>>
>> The ttyS* OMAP serial devices have been renamed to ttyO* in 2.6.37. Replace
>> /dev/ttyS2 with /dev/ttyO2 on your kernel command line (either in the kernel
>> config file if you've activated CONFIG_CMDLINE_FORCE, or in the boot loader
>> environment variables) and you should be able to boot 2.6.38 without any
>> issue. Don't forget to modify /etc/inittab as well.
>>
>> --
>> Regards,
>>
>> Laurent Pinchart
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: OMAP3 ISP deadlocks on my new arm

[Bastian Hecht]

2011/4/16 David Cohen <dacohen@gmail.com>:
> Hi Bastian,
>
> On Thu, Apr 14, 2011 at 1:36 PM, Bastian Hecht <hechtb@googlemail.com> wrote:
>> Yeah!
>>
>> Soooo... when I initialized the the camera (loading a 108 bytes
>> register listing) I just let run the camera and sent images.  So I
>> first realized a counter overflow  if (count++ > 100000) after a few
>> seconds. But this seemed to be handled correctly (ignore and delete
>> HS_VS_IRQ flag) while after 2 or more yavta calls it made the driver
>> hang.
>>
>> I modified my register listing so that the chip gets "booted" silently.
>> In
>> static struct v4l2_subdev_video_ops framix_subdev_video_ops = {
>>        .s_stream       = framix_s_stream, <===============
>> };
>> I correctly check the stream status now and enable/disable the camera signals.
>>
>> I am unsure whether the isp should be able to handle an ongoing data
>> stream independently of ISP_PIPELINE_STREAM_STOPPED.
>
> streamoff should finish synchronously with last ongoing data. So, it
> should have no ongoing data afterwards. Was that your question?

I formulated my reply a bit strange. I meant that that the ongoing
datastream from my camera module (even when the isp-stack is in
stream_stopped state) produces my problem. The question was if it
should be allowed for the camera to send data all time long or only
when it is told to do so by s_stream.

best regards,

Bastian Hecht


> Br,
>
> David Cohen
>
>> If you decide it should, I will gladly help you find out more, just
>> ask. It worked on an OMAP3530 before, but I do not know if it is the
>> hardware or isp software that changed. Unfortunately I can't get an
>> 3530 anymore (I trashed mine...).
>>
>>
>> You helped me so much! Big thanks.
>>
>> Bastian Hecht
>>
>>
>>
>> 2011/4/14 Laurent Pinchart <laurent.pinchart@ideasonboard.com>:
>>> Hi Bastian,
>>>
>>> On Thursday 14 April 2011 10:33:12 Bastian Hecht wrote:
>>>> 2011/4/13 Sakari Ailus <sakari.ailus@maxwell.research.nokia.com>:
>>>> > Bastian Hecht wrote:
>>>> >> Hello people,
>>>> >
>>>> > Hi Bastian,
>>>> >
>>>> > I'm cc'ing Laurent.
>>>> >
>>>> >> I switched to the new DM3730 from IGEP and while it's supposed to be
>>>> >> (almost) the same as the 3530 Version the isp deadlocks
>>>> >> deterministically after I start capturing the second time with yavta.
>>>> >
>>>> > Does the capture work the first time w/o issues?
>>>>
>>>> Yes, I can always run yavta once capturing 4 frames (3 skipped, last
>>>> saved). It usually deadlocks when running yavta the second time but I had
>>>> 1 successful 2nd try (out of 20 maybe).
>>>>
>>>> >> All extra locking debug output is enabled in the kernel .config.
>>>> >
>>>> > I'm not fully certain on what this exactly is that you have below, but
>>>> > it looks like your system is staying in interrupt context all the time.
>>>> > My guess is that the ISP is producing interrupts that the driver is not
>>>> > handling properly, causing the interrupt handler to be called again
>>>> > immediately.
>>>>
>>>> Nice! OK, I'd like to fully understand the panic output, maybe you can
>>>> help there:
>>>> After
>>>> [  376.016906] [<c02e3dc4>] (_raw_spin_unlock_irqrestore+0x40/0x44)
>>>> from [<bf01f678>] (omap3isp_video_queue_streamon+0x80/0x90
>>>> the IRQs get enabled again. Immediately our offending irq wants to get
>>>> served but noone is clearing it.
>>>> At some time, the timer interrupt triggers the watchdog for a kernel panic.
>>>> So the last exception block is the process context that is currently
>>>> active. But why are there 2 irq routines displayed? Is the middle one the
>>>> hardware handling that causes a software irq to be triggered (upper
>>>> block)?
>>>>
>>>> So my next step could be to find the unhandled irq number?
>>>
>>> If the problem is caused by an interrupt storm, the following patch will make
>>> your system responsive again after a couple of seconds (but will kill the ISP
>>> driver :-)). If it doesn't, the problem is likely caused by something else.
>>>
>>> diff --git a/drivers/media/video/omap3isp/isp.c
>>> b/drivers/media/video/omap3isp/isp.c
>>> index de2dec5..6497300 100644
>>> --- a/drivers/media/video/omap3isp/isp.c
>>> +++ b/drivers/media/video/omap3isp/isp.c
>>> @@ -462,6 +464,7 @@ static irqreturn_t isp_isr(int irq, void *_isp)
>>>                                       IRQ0STATUS_CCDC_VD0_IRQ |
>>>                                       IRQ0STATUS_CCDC_VD1_IRQ |
>>>                                       IRQ0STATUS_HS_VS_IRQ;
>>> +       static unsigned int count = 0;
>>>        struct isp_device *isp = _isp;
>>>        u32 irqstatus;
>>>        int ret;
>>> @@ -469,6 +472,11 @@ static irqreturn_t isp_isr(int irq, void *_isp)
>>>        irqstatus = isp_reg_readl(isp, OMAP3_ISP_IOMEM_MAIN, ISP_IRQ0STATUS);
>>>        isp_reg_writel(isp, irqstatus, OMAP3_ISP_IOMEM_MAIN, ISP_IRQ0STATUS);
>>>
>>> +       if (count++ > 100000) {
>>> +               isp_disable_interrupts(isp);
>>> +               return IRQ_HANDLED;
>>> +       }
>>> +
>>>        isp_isr_sbl(isp);
>>>
>>>        if (irqstatus & IRQ0STATUS_CSIA_IRQ) {
>>>
>>>
>>>> > Do you have the same sensor working on OMAP 3530?
>>>>
>>>> I never had this problem on an OMAP 3530, although I better test it
>>>> again with the current setup. I try to get my hands on an 3530 today.
>>>>
>>>> >> I am unsure if this is an ISP thing or a problem in the
>>>> >> interrupthandling software.
>>>> >
>>>> > This has probably something to do with the ISP driver. :-)
>>>> >
>>>> >> The first block is the watchdog that detects the deadlock. The last
>>>> >> block is in the isp-code but how can it hang when trying to UNlock a
>>>> >> spinlock? I am unsure about the 2nd block.
>>>> >> The assembler code of __irq_svc is located in
>>>> >> arch/arm/kernel/entry-armv.S Maybe I should try on
>>>> >> linux-arm@lists.arm.linux.org.uk but I thought I give it a shot here
>>>> >> first.
>>>> >>
>>>> >> I use the omap3isp-2.6.35.3-omap3isp branch from Laurent.
>>>> >
>>>> > Why so old kernel?
>>>>
>>>> I tried a newer version, but there I couldn't get it booting with my
>>>> igep. The kernel unpacked and tried to boot but it froze.
>>>> I decided to use a version that is officially is supported by the igep
>>>> team.
>>>
>>> The ttyS* OMAP serial devices have been renamed to ttyO* in 2.6.37. Replace
>>> /dev/ttyS2 with /dev/ttyO2 on your kernel command line (either in the kernel
>>> config file if you've activated CONFIG_CMDLINE_FORCE, or in the boot loader
>>> environment variables) and you should be able to boot 2.6.38 without any
>>> issue. Don't forget to modify /etc/inittab as well.
>>>
>>> --
>>> Regards,
>>>
>>> Laurent Pinchart
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-media" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>

Re: OMAP3 ISP deadlocks on my new arm

[David Cohen]

On Mon, Apr 18, 2011 at 1:43 PM, Bastian Hecht <hechtb@googlemail.com> wrote:
> 2011/4/16 David Cohen <dacohen@gmail.com>:
>> Hi Bastian,
>>
>> On Thu, Apr 14, 2011 at 1:36 PM, Bastian Hecht <hechtb@googlemail.com> wrote:
>>> Yeah!
>>>
>>> Soooo... when I initialized the the camera (loading a 108 bytes
>>> register listing) I just let run the camera and sent images.  So I
>>> first realized a counter overflow  if (count++ > 100000) after a few
>>> seconds. But this seemed to be handled correctly (ignore and delete
>>> HS_VS_IRQ flag) while after 2 or more yavta calls it made the driver
>>> hang.
>>>
>>> I modified my register listing so that the chip gets "booted" silently.
>>> In
>>> static struct v4l2_subdev_video_ops framix_subdev_video_ops = {
>>>        .s_stream       = framix_s_stream, <===============
>>> };
>>> I correctly check the stream status now and enable/disable the camera signals.
>>>
>>> I am unsure whether the isp should be able to handle an ongoing data
>>> stream independently of ISP_PIPELINE_STREAM_STOPPED.
>>
>> streamoff should finish synchronously with last ongoing data. So, it
>> should have no ongoing data afterwards. Was that your question?
>
> I formulated my reply a bit strange. I meant that that the ongoing
> datastream from my camera module (even when the isp-stack is in
> stream_stopped state) produces my problem. The question was if it
> should be allowed for the camera to send data all time long or only
> when it is told to do so by s_stream.

I may assume you are mentioning a pipeline which includes camera
sensor + ISP. In this case there should be no data.

Regards,

David

>
> best regards,
>
> Bastian Hecht
>
>
>> Br,
>>
>> David Cohen
>>
>>> If you decide it should, I will gladly help you find out more, just
>>> ask. It worked on an OMAP3530 before, but I do not know if it is the
>>> hardware or isp software that changed. Unfortunately I can't get an
>>> 3530 anymore (I trashed mine...).
>>>
>>>
>>> You helped me so much! Big thanks.
>>>
>>> Bastian Hecht
>>>
>>>
>>>
>>> 2011/4/14 Laurent Pinchart <laurent.pinchart@ideasonboard.com>:
>>>> Hi Bastian,
>>>>
>>>> On Thursday 14 April 2011 10:33:12 Bastian Hecht wrote:
>>>>> 2011/4/13 Sakari Ailus <sakari.ailus@maxwell.research.nokia.com>:
>>>>> > Bastian Hecht wrote:
>>>>> >> Hello people,
>>>>> >
>>>>> > Hi Bastian,
>>>>> >
>>>>> > I'm cc'ing Laurent.
>>>>> >
>>>>> >> I switched to the new DM3730 from IGEP and while it's supposed to be
>>>>> >> (almost) the same as the 3530 Version the isp deadlocks
>>>>> >> deterministically after I start capturing the second time with yavta.
>>>>> >
>>>>> > Does the capture work the first time w/o issues?
>>>>>
>>>>> Yes, I can always run yavta once capturing 4 frames (3 skipped, last
>>>>> saved). It usually deadlocks when running yavta the second time but I had
>>>>> 1 successful 2nd try (out of 20 maybe).
>>>>>
>>>>> >> All extra locking debug output is enabled in the kernel .config.
>>>>> >
>>>>> > I'm not fully certain on what this exactly is that you have below, but
>>>>> > it looks like your system is staying in interrupt context all the time.
>>>>> > My guess is that the ISP is producing interrupts that the driver is not
>>>>> > handling properly, causing the interrupt handler to be called again
>>>>> > immediately.
>>>>>
>>>>> Nice! OK, I'd like to fully understand the panic output, maybe you can
>>>>> help there:
>>>>> After
>>>>> [  376.016906] [<c02e3dc4>] (_raw_spin_unlock_irqrestore+0x40/0x44)
>>>>> from [<bf01f678>] (omap3isp_video_queue_streamon+0x80/0x90
>>>>> the IRQs get enabled again. Immediately our offending irq wants to get
>>>>> served but noone is clearing it.
>>>>> At some time, the timer interrupt triggers the watchdog for a kernel panic.
>>>>> So the last exception block is the process context that is currently
>>>>> active. But why are there 2 irq routines displayed? Is the middle one the
>>>>> hardware handling that causes a software irq to be triggered (upper
>>>>> block)?
>>>>>
>>>>> So my next step could be to find the unhandled irq number?
>>>>
>>>> If the problem is caused by an interrupt storm, the following patch will make
>>>> your system responsive again after a couple of seconds (but will kill the ISP
>>>> driver :-)). If it doesn't, the problem is likely caused by something else.
>>>>
>>>> diff --git a/drivers/media/video/omap3isp/isp.c
>>>> b/drivers/media/video/omap3isp/isp.c
>>>> index de2dec5..6497300 100644
>>>> --- a/drivers/media/video/omap3isp/isp.c
>>>> +++ b/drivers/media/video/omap3isp/isp.c
>>>> @@ -462,6 +464,7 @@ static irqreturn_t isp_isr(int irq, void *_isp)
>>>>                                       IRQ0STATUS_CCDC_VD0_IRQ |
>>>>                                       IRQ0STATUS_CCDC_VD1_IRQ |
>>>>                                       IRQ0STATUS_HS_VS_IRQ;
>>>> +       static unsigned int count = 0;
>>>>        struct isp_device *isp = _isp;
>>>>        u32 irqstatus;
>>>>        int ret;
>>>> @@ -469,6 +472,11 @@ static irqreturn_t isp_isr(int irq, void *_isp)
>>>>        irqstatus = isp_reg_readl(isp, OMAP3_ISP_IOMEM_MAIN, ISP_IRQ0STATUS);
>>>>        isp_reg_writel(isp, irqstatus, OMAP3_ISP_IOMEM_MAIN, ISP_IRQ0STATUS);
>>>>
>>>> +       if (count++ > 100000) {
>>>> +               isp_disable_interrupts(isp);
>>>> +               return IRQ_HANDLED;
>>>> +       }
>>>> +
>>>>        isp_isr_sbl(isp);
>>>>
>>>>        if (irqstatus & IRQ0STATUS_CSIA_IRQ) {
>>>>
>>>>
>>>>> > Do you have the same sensor working on OMAP 3530?
>>>>>
>>>>> I never had this problem on an OMAP 3530, although I better test it
>>>>> again with the current setup. I try to get my hands on an 3530 today.
>>>>>
>>>>> >> I am unsure if this is an ISP thing or a problem in the
>>>>> >> interrupthandling software.
>>>>> >
>>>>> > This has probably something to do with the ISP driver. :-)
>>>>> >
>>>>> >> The first block is the watchdog that detects the deadlock. The last
>>>>> >> block is in the isp-code but how can it hang when trying to UNlock a
>>>>> >> spinlock? I am unsure about the 2nd block.
>>>>> >> The assembler code of __irq_svc is located in
>>>>> >> arch/arm/kernel/entry-armv.S Maybe I should try on
>>>>> >> linux-arm@lists.arm.linux.org.uk but I thought I give it a shot here
>>>>> >> first.
>>>>> >>
>>>>> >> I use the omap3isp-2.6.35.3-omap3isp branch from Laurent.
>>>>> >
>>>>> > Why so old kernel?
>>>>>
>>>>> I tried a newer version, but there I couldn't get it booting with my
>>>>> igep. The kernel unpacked and tried to boot but it froze.
>>>>> I decided to use a version that is officially is supported by the igep
>>>>> team.
>>>>
>>>> The ttyS* OMAP serial devices have been renamed to ttyO* in 2.6.37. Replace
>>>> /dev/ttyS2 with /dev/ttyO2 on your kernel command line (either in the kernel
>>>> config file if you've activated CONFIG_CMDLINE_FORCE, or in the boot loader
>>>> environment variables) and you should be able to boot 2.6.38 without any
>>>> issue. Don't forget to modify /etc/inittab as well.
>>>>
>>>> --
>>>> Regards,
>>>>
>>>> Laurent Pinchart
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-media" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
>

Re: OMAP3 ISP deadlocks on my new arm

[Laurent Pinchart]

On Monday 18 April 2011 16:17:15 David Cohen wrote:
> On Mon, Apr 18, 2011 at 1:43 PM, Bastian Hecht wrote:
> > 2011/4/16 David Cohen:
> >> On Thu, Apr 14, 2011 at 1:36 PM, Bastian Hecht wrote:
> >>> Yeah!
> >>> 
> >>> Soooo... when I initialized the the camera (loading a 108 bytes
> >>> register listing) I just let run the camera and sent images.  So I
> >>> first realized a counter overflow  if (count++ > 100000) after a few
> >>> seconds. But this seemed to be handled correctly (ignore and delete
> >>> HS_VS_IRQ flag) while after 2 or more yavta calls it made the driver
> >>> hang.
> >>> 
> >>> I modified my register listing so that the chip gets "booted" silently.
> >>> In
> >>> static struct v4l2_subdev_video_ops framix_subdev_video_ops = {
> >>>        .s_stream       = framix_s_stream, <===============
> >>> };
> >>> I correctly check the stream status now and enable/disable the camera
> >>> signals.
> >>> 
> >>> I am unsure whether the isp should be able to handle an ongoing data
> >>> stream independently of ISP_PIPELINE_STREAM_STOPPED.
> >> 
> >> streamoff should finish synchronously with last ongoing data. So, it
> >> should have no ongoing data afterwards. Was that your question?
> > 
> > I formulated my reply a bit strange. I meant that that the ongoing
> > datastream from my camera module (even when the isp-stack is in
> > stream_stopped state) produces my problem. The question was if it
> > should be allowed for the camera to send data all time long or only
> > when it is told to do so by s_stream.
> 
> I may assume you are mentioning a pipeline which includes camera
> sensor + ISP. In this case there should be no data.

That's the ideal situation: sensors should not produce any data (or rather any 
transition on the VS/HS signals) when they're supposed to be stopped. 
Unfortunately that's not always easy, as some dumb sensors (or sensor-like 
hardware) can't be stopped. The ISP driver should be able to cope with that in 
a way that doesn't kill the system completely.

I've noticed the same issue with a Caspa camera module and an OMAP3503-based 
Gumstix. I'll try to come up with a good fix.

-- 
Regards,

Laurent Pinchart

Re: OMAP3 ISP deadlocks on my new arm

[Sakari Ailus]

Laurent Pinchart wrote:
...
> That's the ideal situation: sensors should not produce any data (or rather any 
> transition on the VS/HS signals) when they're supposed to be stopped. 
> Unfortunately that's not always easy, as some dumb sensors (or sensor-like 
> hardware) can't be stopped. The ISP driver should be able to cope with that in 
> a way that doesn't kill the system completely.
> 
> I've noticed the same issue with a Caspa camera module and an OMAP3503-based 
> Gumstix. I'll try to come up with a good fix.

Hi Laurent, others,

Do you think the cause for this is that the system is jammed in handling
HS_VS interrupts triggered for every HS?

A quick fix for this could be just choosing either VS configuration when
configuring the CCDC. Alternatively, HS_VS interrupts could be just
disabled until omap3isp_configure_interface().

But as the sensor is sending images all the time, proper VS
configuration would be needed, or the counting of lines in the CCDC (VD*
interrupts) is affected as well. The VD0 interrupt, which is used to
trigger an interrupt near the end of the frame, may be triggered one
line too early on the first frame, or too late. But this is up to a
configuration. I don't think it's a real issue to trigger it one line
too early.

Anything else?

Cheers,

-- 
Sakari Ailus
sakari.ailus@maxwell.research.nokia.com

Re: OMAP3 ISP deadlocks on my new arm

[Laurent Pinchart]

Hi Sakari,

On Tuesday 19 April 2011 09:31:05 Sakari Ailus wrote:
> Laurent Pinchart wrote:
> ...
> 
> > That's the ideal situation: sensors should not produce any data (or
> > rather any transition on the VS/HS signals) when they're supposed to be
> > stopped. Unfortunately that's not always easy, as some dumb sensors (or
> > sensor-like hardware) can't be stopped. The ISP driver should be able to
> > cope with that in a way that doesn't kill the system completely.
> > 
> > I've noticed the same issue with a Caspa camera module and an
> > OMAP3503-based Gumstix. I'll try to come up with a good fix.
> 
> Hi Laurent, others,
> 
> Do you think the cause for this is that the system is jammed in handling
> HS_VS interrupts triggered for every HS?

That was my initial guess, yes.

> A quick fix for this could be just choosing either VS configuration when
> configuring the CCDC. Alternatively, HS_VS interrupts could be just
> disabled until omap3isp_configure_interface().
> 
> But as the sensor is sending images all the time, proper VS configuration
> would be needed, or the counting of lines in the CCDC (VD* interrupts) is
> affected as well. The VD0 interrupt, which is used to trigger an interrupt
> near the end of the frame, may be triggered one line too early on the first
> frame, or too late. But this is up to a configuration. I don't think it's a
> real issue to trigger it one line too early.
> 
> Anything else?

I've tried delaying the HS_VS interrupt enable to the CCDC configuration 
function, after configuring the bridge (and thus the HS/VS interrupt source 
selection). To my surprise it didn't fix the problem, I still get tons of 
HS_VS interrupts (100000 in about 2.6 seconds) that kill the system.

I'll need to hook a scope to the HS and VS signals.

-- 
Regards,

Laurent Pinchart

Re: OMAP3 ISP deadlocks on my new arm

[Bastian Hecht]

2011/4/21 Laurent Pinchart <laurent.pinchart@ideasonboard.com>:
> Hi Sakari,
>
> On Tuesday 19 April 2011 09:31:05 Sakari Ailus wrote:
>> Laurent Pinchart wrote:
>> ...
>>
>> > That's the ideal situation: sensors should not produce any data (or
>> > rather any transition on the VS/HS signals) when they're supposed to be
>> > stopped. Unfortunately that's not always easy, as some dumb sensors (or
>> > sensor-like hardware) can't be stopped. The ISP driver should be able to
>> > cope with that in a way that doesn't kill the system completely.
>> >
>> > I've noticed the same issue with a Caspa camera module and an
>> > OMAP3503-based Gumstix. I'll try to come up with a good fix.
>>
>> Hi Laurent, others,
>>
>> Do you think the cause for this is that the system is jammed in handling
>> HS_VS interrupts triggered for every HS?
>
> That was my initial guess, yes.
>
>> A quick fix for this could be just choosing either VS configuration when
>> configuring the CCDC. Alternatively, HS_VS interrupts could be just
>> disabled until omap3isp_configure_interface().
>>
>> But as the sensor is sending images all the time, proper VS configuration
>> would be needed, or the counting of lines in the CCDC (VD* interrupts) is
>> affected as well. The VD0 interrupt, which is used to trigger an interrupt
>> near the end of the frame, may be triggered one line too early on the first
>> frame, or too late. But this is up to a configuration. I don't think it's a
>> real issue to trigger it one line too early.
>>
>> Anything else?

Hello Laurent,

> I've tried delaying the HS_VS interrupt enable to the CCDC configuration
> function, after configuring the bridge (and thus the HS/VS interrupt source
> selection). To my surprise it didn't fix the problem, I still get tons of
> HS_VS interrupts (100000 in about 2.6 seconds) that kill the system.
>
> I'll need to hook a scope to the HS and VS signals.

have you worked on this problem? Today in my setup I took a longer
cable and ran again into the hs/vs interrupt storm (it still works
with a short cable).
I can tackle this issue too, but to avoid double work I wanted to ask
if you worked out something in the meantime.

Best regards,

 Bastian Hecht

>
> --
> Regards,
>
> Laurent Pinchart
>

Re: OMAP3 ISP deadlocks on my new arm

[Laurent Pinchart]

Hi Bastian,

On Tuesday 26 April 2011 17:39:41 Bastian Hecht wrote:
> 2011/4/21 Laurent Pinchart <laurent.pinchart@ideasonboard.com>:
> > On Tuesday 19 April 2011 09:31:05 Sakari Ailus wrote:
> >> Laurent Pinchart wrote:
> >> ...
> >> 
> >> > That's the ideal situation: sensors should not produce any data (or
> >> > rather any transition on the VS/HS signals) when they're supposed to
> >> > be stopped. Unfortunately that's not always easy, as some dumb
> >> > sensors (or sensor-like hardware) can't be stopped. The ISP driver
> >> > should be able to cope with that in a way that doesn't kill the
> >> > system completely.
> >> > 
> >> > I've noticed the same issue with a Caspa camera module and an
> >> > OMAP3503-based Gumstix. I'll try to come up with a good fix.
> >> 
> >> Hi Laurent, others,
> >> 
> >> Do you think the cause for this is that the system is jammed in handling
> >> HS_VS interrupts triggered for every HS?
> > 
> > That was my initial guess, yes.
> > 
> >> A quick fix for this could be just choosing either VS configuration when
> >> configuring the CCDC. Alternatively, HS_VS interrupts could be just
> >> disabled until omap3isp_configure_interface().
> >> 
> >> But as the sensor is sending images all the time, proper VS
> >> configuration would be needed, or the counting of lines in the CCDC
> >> (VD* interrupts) is affected as well. The VD0 interrupt, which is used
> >> to trigger an interrupt near the end of the frame, may be triggered one
> >> line too early on the first frame, or too late. But this is up to a
> >> configuration. I don't think it's a real issue to trigger it one line
> >> too early.
> >> 
> >> Anything else?
> 
> Hello Laurent,
> 
> > I've tried delaying the HS_VS interrupt enable to the CCDC configuration
> > function, after configuring the bridge (and thus the HS/VS interrupt
> > source selection). To my surprise it didn't fix the problem, I still get
> > tons of HS_VS interrupts (100000 in about 2.6 seconds) that kill the
> > system.
> > 
> > I'll need to hook a scope to the HS and VS signals.
> 
> have you worked on this problem? Today in my setup I took a longer cable and
> ran again into the hs/vs interrupt storm (it still works with a short
> cable).
> I can tackle this issue too, but to avoid double work I wanted to ask if you
> worked out something in the meantime.

In my case the issue was caused by a combination of two hardware design 
mistakes. The first one was to use a TXB0801 chip to translate the 3.3V sensor 
levels to the 1.8V OMAP levels. The TXB0801 4kΩ output impedance, combined 
with the OMAP3 100µA pull-ups on the HS and VS signals, produces a ~400mV 
voltage for low logic levels.

Then, the XCLKA signal is next to the VS signal on the cable connecting the 
camera module to the OMAP board. When XCLKA is turned on, cross-talk produces 
a 400mV peak-to-peak noise on the VS signal.

The combination of those two effects create a noisy VS signal that crosses the 
OMAP3 input level detection gap at high frequency, leading to an interrupt 
storm. The workaround is to disable the pull-ups on the HS and VS signals, the 
solution is to redesign the hardware to replace the level translators and 
reorganize signals on the camera module cable.

Is your situation any similar ?

-- 
Regards,

Laurent Pinchart

Re: OMAP3 ISP deadlocks on my new arm

[Bastian Hecht]

2011/4/26 Laurent Pinchart <laurent.pinchart@ideasonboard.com>:
> Hi Bastian,
>
> On Tuesday 26 April 2011 17:39:41 Bastian Hecht wrote:
>> 2011/4/21 Laurent Pinchart <laurent.pinchart@ideasonboard.com>:
>> > On Tuesday 19 April 2011 09:31:05 Sakari Ailus wrote:
>> >> Laurent Pinchart wrote:
>> >> ...
>> >>
>> >> > That's the ideal situation: sensors should not produce any data (or
>> >> > rather any transition on the VS/HS signals) when they're supposed to
>> >> > be stopped. Unfortunately that's not always easy, as some dumb
>> >> > sensors (or sensor-like hardware) can't be stopped. The ISP driver
>> >> > should be able to cope with that in a way that doesn't kill the
>> >> > system completely.
>> >> >
>> >> > I've noticed the same issue with a Caspa camera module and an
>> >> > OMAP3503-based Gumstix. I'll try to come up with a good fix.
>> >>
>> >> Hi Laurent, others,
>> >>
>> >> Do you think the cause for this is that the system is jammed in handling
>> >> HS_VS interrupts triggered for every HS?
>> >
>> > That was my initial guess, yes.
>> >
>> >> A quick fix for this could be just choosing either VS configuration when
>> >> configuring the CCDC. Alternatively, HS_VS interrupts could be just
>> >> disabled until omap3isp_configure_interface().
>> >>
>> >> But as the sensor is sending images all the time, proper VS
>> >> configuration would be needed, or the counting of lines in the CCDC
>> >> (VD* interrupts) is affected as well. The VD0 interrupt, which is used
>> >> to trigger an interrupt near the end of the frame, may be triggered one
>> >> line too early on the first frame, or too late. But this is up to a
>> >> configuration. I don't think it's a real issue to trigger it one line
>> >> too early.
>> >>
>> >> Anything else?
>>
>> Hello Laurent,
>>
>> > I've tried delaying the HS_VS interrupt enable to the CCDC configuration
>> > function, after configuring the bridge (and thus the HS/VS interrupt
>> > source selection). To my surprise it didn't fix the problem, I still get
>> > tons of HS_VS interrupts (100000 in about 2.6 seconds) that kill the
>> > system.
>> >
>> > I'll need to hook a scope to the HS and VS signals.
>>
>> have you worked on this problem? Today in my setup I took a longer cable and
>> ran again into the hs/vs interrupt storm (it still works with a short
>> cable).
>> I can tackle this issue too, but to avoid double work I wanted to ask if you
>> worked out something in the meantime.


> In my case the issue was caused by a combination of two hardware design
> mistakes. The first one was to use a TXB0801 chip to translate the 3.3V sensor
> levels to the 1.8V OMAP levels. The TXB0801 4kΩ output impedance, combined
> with the OMAP3 100µA pull-ups on the HS and VS signals, produces a ~400mV
> voltage for low logic levels.
>
> Then, the XCLKA signal is next to the VS signal on the cable connecting the
> camera module to the OMAP board. When XCLKA is turned on, cross-talk produces
> a 400mV peak-to-peak noise on the VS signal.
>
> The combination of those two effects create a noisy VS signal that crosses the
> OMAP3 input level detection gap at high frequency, leading to an interrupt
> storm. The workaround is to disable the pull-ups on the HS and VS signals, the
> solution is to redesign the hardware to replace the level translators and
> reorganize signals on the camera module cable.

Hi Laurent,

> Is your situation any similar ?

The long data line (~35cm now at 24MHz) certainly can have an impact
but I haven't measured any crosstalk so far. But I'm on another trail
now. I found out that on my board the interrupt line is shared with
 24:          0        INTC  omap-iommu.0

Is the following scenario possible?

1. The omap-iommu isr is registered
2. The isp gets set up (it enables interrupts and disables them again
at the end of the probe function)
3. Later I activate the xclk from within my driver
  3a. isp_set_xclk() gets the lock omap3isp_get(isp) and so
enable_interrupts() is called
  3b. The new xclk on my chip makes my hardware create a hs/vs int
(either crosstalk, another hardware bug like yours, or simply my chip
sends a spurious interrupt for any reason)
  3c.  isp_set_xclk() puts the lock omap3isp_put(isp) and so
disable_interrupts() is called

Can there exist a race condition between the omap3isp raising the
interrupt pin before 3c or after 3c?

If after 3c the omap-iommu isr loops forever as the omap3isp int flag
is never cleared.

I keep debbuging and trying to find further clues.

Best regards,

Bastian


> --
> Regards,
>
> Laurent Pinchart
>

Re: OMAP3 ISP deadlocks on my new arm

[Bastian Hecht]

2011/4/27 Bastian Hecht <hechtb@googlemail.com>:
> 2011/4/26 Laurent Pinchart <laurent.pinchart@ideasonboard.com>:
>> Hi Bastian,
>>
>> On Tuesday 26 April 2011 17:39:41 Bastian Hecht wrote:
>>> 2011/4/21 Laurent Pinchart <laurent.pinchart@ideasonboard.com>:
>>> > On Tuesday 19 April 2011 09:31:05 Sakari Ailus wrote:
>>> >> Laurent Pinchart wrote:
>>> >> ...
>>> >>
>>> >> > That's the ideal situation: sensors should not produce any data (or
>>> >> > rather any transition on the VS/HS signals) when they're supposed to
>>> >> > be stopped. Unfortunately that's not always easy, as some dumb
>>> >> > sensors (or sensor-like hardware) can't be stopped. The ISP driver
>>> >> > should be able to cope with that in a way that doesn't kill the
>>> >> > system completely.
>>> >> >
>>> >> > I've noticed the same issue with a Caspa camera module and an
>>> >> > OMAP3503-based Gumstix. I'll try to come up with a good fix.
>>> >>
>>> >> Hi Laurent, others,
>>> >>
>>> >> Do you think the cause for this is that the system is jammed in handling
>>> >> HS_VS interrupts triggered for every HS?
>>> >
>>> > That was my initial guess, yes.
>>> >
>>> >> A quick fix for this could be just choosing either VS configuration when
>>> >> configuring the CCDC. Alternatively, HS_VS interrupts could be just
>>> >> disabled until omap3isp_configure_interface().
>>> >>
>>> >> But as the sensor is sending images all the time, proper VS
>>> >> configuration would be needed, or the counting of lines in the CCDC
>>> >> (VD* interrupts) is affected as well. The VD0 interrupt, which is used
>>> >> to trigger an interrupt near the end of the frame, may be triggered one
>>> >> line too early on the first frame, or too late. But this is up to a
>>> >> configuration. I don't think it's a real issue to trigger it one line
>>> >> too early.
>>> >>
>>> >> Anything else?
>>>
>>> Hello Laurent,
>>>
>>> > I've tried delaying the HS_VS interrupt enable to the CCDC configuration
>>> > function, after configuring the bridge (and thus the HS/VS interrupt
>>> > source selection). To my surprise it didn't fix the problem, I still get
>>> > tons of HS_VS interrupts (100000 in about 2.6 seconds) that kill the
>>> > system.
>>> >
>>> > I'll need to hook a scope to the HS and VS signals.
>>>
>>> have you worked on this problem? Today in my setup I took a longer cable and
>>> ran again into the hs/vs interrupt storm (it still works with a short
>>> cable).
>>> I can tackle this issue too, but to avoid double work I wanted to ask if you
>>> worked out something in the meantime.
>
>
>> In my case the issue was caused by a combination of two hardware design
>> mistakes. The first one was to use a TXB0801 chip to translate the 3.3V sensor
>> levels to the 1.8V OMAP levels. The TXB0801 4kΩ output impedance, combined
>> with the OMAP3 100µA pull-ups on the HS and VS signals, produces a ~400mV
>> voltage for low logic levels.
>>
>> Then, the XCLKA signal is next to the VS signal on the cable connecting the
>> camera module to the OMAP board. When XCLKA is turned on, cross-talk produces
>> a 400mV peak-to-peak noise on the VS signal.
>>
>> The combination of those two effects create a noisy VS signal that crosses the
>> OMAP3 input level detection gap at high frequency, leading to an interrupt
>> storm. The workaround is to disable the pull-ups on the HS and VS signals, the
>> solution is to redesign the hardware to replace the level translators and
>> reorganize signals on the camera module cable.
>
> Hi Laurent,
>
>> Is your situation any similar ?
>
> The long data line (~35cm now at 24MHz) certainly can have an impact
> but I haven't measured any crosstalk so far. But I'm on another trail
> now. I found out that on my board the interrupt line is shared with
>  24:          0        INTC  omap-iommu.0
>
> Is the following scenario possible?
>
> 1. The omap-iommu isr is registered
> 2. The isp gets set up (it enables interrupts and disables them again
> at the end of the probe function)
> 3. Later I activate the xclk from within my driver
>  3a. isp_set_xclk() gets the lock omap3isp_get(isp) and so
> enable_interrupts() is called
>  3b. The new xclk on my chip makes my hardware create a hs/vs int
> (either crosstalk, another hardware bug like yours, or simply my chip
> sends a spurious interrupt for any reason)
>  3c.  isp_set_xclk() puts the lock omap3isp_put(isp) and so
> disable_interrupts() is called
>
> Can there exist a race condition between the omap3isp raising the
> interrupt pin before 3c or after 3c?

Argh... I oversaw that the omap3isp isr handler stays registered all
time long so the theory is wrong.

> If after 3c the omap-iommu isr loops forever as the omap3isp int flag
> is never cleared.
>
> I keep debbuging and trying to find further clues.
>
> Best regards,
>
> Bastian
>
>
>> --
>> Regards,
>>
>> Laurent Pinchart
>>
>

Re: OMAP3 ISP deadlocks on my new arm

[Laurent Pinchart]

Hi Bastian,

On Wednesday 27 April 2011 12:55:24 Bastian Hecht wrote:
> 2011/4/27 Bastian Hecht <hechtb@googlemail.com>:
> > 2011/4/26 Laurent Pinchart <laurent.pinchart@ideasonboard.com>:
> >> On Tuesday 26 April 2011 17:39:41 Bastian Hecht wrote:
> >>> 2011/4/21 Laurent Pinchart <laurent.pinchart@ideasonboard.com>:
> >>> > On Tuesday 19 April 2011 09:31:05 Sakari Ailus wrote:
> >>> >> Laurent Pinchart wrote:
> >>> >> ...
> >>> >> 
> >>> >> > That's the ideal situation: sensors should not produce any data
> >>> >> > (or rather any transition on the VS/HS signals) when they're
> >>> >> > supposed to be stopped. Unfortunately that's not always easy, as
> >>> >> > some dumb sensors (or sensor-like hardware) can't be stopped. The
> >>> >> > ISP driver should be able to cope with that in a way that doesn't
> >>> >> > kill the system completely.
> >>> >> > 
> >>> >> > I've noticed the same issue with a Caspa camera module and an
> >>> >> > OMAP3503-based Gumstix. I'll try to come up with a good fix.
> >>> >> 
> >>> >> Hi Laurent, others,
> >>> >> 
> >>> >> Do you think the cause for this is that the system is jammed in
> >>> >> handling HS_VS interrupts triggered for every HS?
> >>> > 
> >>> > That was my initial guess, yes.
> >>> > 
> >>> >> A quick fix for this could be just choosing either VS configuration
> >>> >> when configuring the CCDC. Alternatively, HS_VS interrupts could be
> >>> >> just disabled until omap3isp_configure_interface().
> >>> >> 
> >>> >> But as the sensor is sending images all the time, proper VS
> >>> >> configuration would be needed, or the counting of lines in the CCDC
> >>> >> (VD* interrupts) is affected as well. The VD0 interrupt, which is
> >>> >> used to trigger an interrupt near the end of the frame, may be
> >>> >> triggered one line too early on the first frame, or too late. But
> >>> >> this is up to a configuration. I don't think it's a real issue to
> >>> >> trigger it one line too early.
> >>> >> 
> >>> >> Anything else?
> >>> 
> >>> Hello Laurent,
> >>> 
> >>> > I've tried delaying the HS_VS interrupt enable to the CCDC
> >>> > configuration function, after configuring the bridge (and thus the
> >>> > HS/VS interrupt source selection). To my surprise it didn't fix the
> >>> > problem, I still get tons of HS_VS interrupts (100000 in about 2.6
> >>> > seconds) that kill the system.
> >>> > 
> >>> > I'll need to hook a scope to the HS and VS signals.
> >>> 
> >>> have you worked on this problem? Today in my setup I took a longer
> >>> cable and ran again into the hs/vs interrupt storm (it still works
> >>> with a short cable).
> >>> I can tackle this issue too, but to avoid double work I wanted to ask
> >>> if you worked out something in the meantime.
> >> 
> >> In my case the issue was caused by a combination of two hardware design
> >> mistakes. The first one was to use a TXB0801 chip to translate the 3.3V
> >> sensor levels to the 1.8V OMAP levels. The TXB0801 4kΩ output
> >> impedance, combined with the OMAP3 100µA pull-ups on the HS and VS
> >> signals, produces a ~400mV voltage for low logic levels.
> >> 
> >> Then, the XCLKA signal is next to the VS signal on the cable connecting
> >> the camera module to the OMAP board. When XCLKA is turned on,
> >> cross-talk produces a 400mV peak-to-peak noise on the VS signal.
> >> 
> >> The combination of those two effects create a noisy VS signal that
> >> crosses the OMAP3 input level detection gap at high frequency, leading
> >> to an interrupt storm. The workaround is to disable the pull-ups on the
> >> HS and VS signals, the solution is to redesign the hardware to replace
> >> the level translators and reorganize signals on the camera module
> >> cable.
> > 
> > Hi Laurent,
> > 
> >> Is your situation any similar ?
> > 
> > The long data line (~35cm now at 24MHz) certainly can have an impact
> > but I haven't measured any crosstalk so far. But I'm on another trail
> > now. I found out that on my board the interrupt line is shared with
> >  24:          0        INTC  omap-iommu.0
> > 
> > Is the following scenario possible?
> > 
> > 1. The omap-iommu isr is registered
> > 2. The isp gets set up (it enables interrupts and disables them again
> > at the end of the probe function)
> > 3. Later I activate the xclk from within my driver
> >  3a. isp_set_xclk() gets the lock omap3isp_get(isp) and so
> > enable_interrupts() is called
> >  3b. The new xclk on my chip makes my hardware create a hs/vs int
> > (either crosstalk, another hardware bug like yours, or simply my chip
> > sends a spurious interrupt for any reason)
> >  3c.  isp_set_xclk() puts the lock omap3isp_put(isp) and so
> > disable_interrupts() is called
> > 
> > Can there exist a race condition between the omap3isp raising the
> > interrupt pin before 3c or after 3c?
> 
> Argh... I oversaw that the omap3isp isr handler stays registered all
> time long so the theory is wrong.

No luck :-)

The first investigation step is to check which interrupt source causes the
interrupts storm. The following test patch should help.

diff --git a/drivers/media/video/omap3isp/isp.c b/drivers/media/video/omap3isp/isp.c
index de2dec5..c4e6455 100644
--- a/drivers/media/video/omap3isp/isp.c
+++ b/drivers/media/video/omap3isp/isp.c
@@ -400,6 +400,38 @@ static inline void isp_isr_dbg(struct isp_device *isp, u32 irqstatus)
 	printk(KERN_CONT "\n");
 }
 
+static unsigned int isp_isr_count[32];
+
+static inline void isp_isr_account(struct isp_device *isp, u32 irqstatus)
+{
+	unsigned int i;
+
+	spin_lock(&isp->isr_account_lock);
+	for (i = 0; i < 32; i++) {
+		if (irqstatus & (1 << i))
+			isp_isr_count[i]++;
+	}
+	spin_unlock(&isp->isr_account_lock);
+}
+
+static ssize_t isp_isr_account_show(struct device *dev,
+	struct device_attribute *attr, char *buf)
+{
+	struct isp_device *isp = container_of(to_media_device(to_media_devnode(dev)), struct isp_device, media_dev);
+	unsigned long flags;
+	unsigned int i;
+	int ret = 0;
+
+	spin_lock_irqsave(&isp->isr_account_lock, flags);
+	for (i = 0; i < 32; i++)
+		ret += sprintf(buf + ret, "%u\t%u\n", i, isp_isr_count[i]);
+	spin_unlock_irqrestore(&isp->isr_account_lock, flags);
+
+	return ret;
+}
+
+static DEVICE_ATTR(isr_account, S_IRUGO, isp_isr_account_show, NULL);
+
 static void isp_isr_sbl(struct isp_device *isp)
 {
 	struct device *dev = isp->dev;
@@ -462,6 +494,7 @@ static irqreturn_t isp_isr(int irq, void *_isp)
 				       IRQ0STATUS_CCDC_VD0_IRQ |
 				       IRQ0STATUS_CCDC_VD1_IRQ |
 				       IRQ0STATUS_HS_VS_IRQ;
+	static unsigned int count = 0;
 	struct isp_device *isp = _isp;
 	u32 irqstatus;
 	int ret;
@@ -469,6 +502,10 @@ static irqreturn_t isp_isr(int irq, void *_isp)
 	irqstatus = isp_reg_readl(isp, OMAP3_ISP_IOMEM_MAIN, ISP_IRQ0STATUS);
 	isp_reg_writel(isp, irqstatus, OMAP3_ISP_IOMEM_MAIN, ISP_IRQ0STATUS);
 
+	isp_isr_account(isp, irqstatus);
+	if (count++ >= 100000)
+		isp_disable_interrupts(isp);
+
 	isp_isr_sbl(isp);
 
 	if (irqstatus & IRQ0STATUS_CSIA_IRQ) {
@@ -1971,6 +2008,7 @@ static int isp_remove(struct platform_device *pdev)
 	struct isp_device *isp = platform_get_drvdata(pdev);
 	int i;
 
+	device_remove_file(&isp->media_dev.devnode.dev, &dev_attr_isr_account);
 	isp_unregister_entities(isp);
 	isp_cleanup_modules(isp);
 
@@ -2067,6 +2105,7 @@ static int isp_probe(struct platform_device *pdev)
 
 	mutex_init(&isp->isp_mutex);
 	spin_lock_init(&isp->stat_lock);
+	spin_lock_init(&isp->isr_account_lock);
 
 	isp->dev = &pdev->dev;
 	isp->pdata = pdata;
@@ -2156,6 +2195,8 @@ static int isp_probe(struct platform_device *pdev)
 	isp_power_settings(isp, 1);
 	omap3isp_put(isp);
 
+	ret = device_create_file(&isp->media_dev.devnode.dev, &dev_attr_isr_account);
+
 	return 0;
 
 error_modules:
diff --git a/drivers/media/video/omap3isp/isp.h b/drivers/media/video/omap3isp/isp.h
index 2620c40..b3f8448 100644
--- a/drivers/media/video/omap3isp/isp.h
+++ b/drivers/media/video/omap3isp/isp.h
@@ -259,6 +259,7 @@ struct isp_device {
 
 	/* ISP Obj */
 	spinlock_t stat_lock;	/* common lock for statistic drivers */
+	spinlock_t isr_account_lock;
 	struct mutex isp_mutex;	/* For handling ref_count field */
 	bool needs_reset;
 	int has_context;

Start a capture, wait a couple of settings for ISP interrupts to get disabled,
and cat the isr_account sysfs file
(/sys/bus/platform/devices/omap3isp/media0/isr_account if I remember
correctly). My guess is that you will get approximatively 100000 HS_VS
interrupts (31). Let's then move on from there after you've confirmed that the
guess is correct.

-- 
Regards,

Laurent Pinchart

Re: OMAP3 ISP deadlocks on my new arm

[Bastian Hecht]

2011/4/27 Laurent Pinchart <laurent.pinchart@ideasonboard.com>:
> Hi Bastian,
>
> On Wednesday 27 April 2011 12:55:24 Bastian Hecht wrote:
>> 2011/4/27 Bastian Hecht <hechtb@googlemail.com>:
>> > 2011/4/26 Laurent Pinchart <laurent.pinchart@ideasonboard.com>:
>> >> On Tuesday 26 April 2011 17:39:41 Bastian Hecht wrote:
>> >>> 2011/4/21 Laurent Pinchart <laurent.pinchart@ideasonboard.com>:
>> >>> > On Tuesday 19 April 2011 09:31:05 Sakari Ailus wrote:
>> >>> >> Laurent Pinchart wrote:
>> >>> >> ...
>> >>> >>
>> >>> >> > That's the ideal situation: sensors should not produce any data
>> >>> >> > (or rather any transition on the VS/HS signals) when they're
>> >>> >> > supposed to be stopped. Unfortunately that's not always easy, as
>> >>> >> > some dumb sensors (or sensor-like hardware) can't be stopped. The
>> >>> >> > ISP driver should be able to cope with that in a way that doesn't
>> >>> >> > kill the system completely.
>> >>> >> >
>> >>> >> > I've noticed the same issue with a Caspa camera module and an
>> >>> >> > OMAP3503-based Gumstix. I'll try to come up with a good fix.
>> >>> >>
>> >>> >> Hi Laurent, others,
>> >>> >>
>> >>> >> Do you think the cause for this is that the system is jammed in
>> >>> >> handling HS_VS interrupts triggered for every HS?
>> >>> >
>> >>> > That was my initial guess, yes.
>> >>> >
>> >>> >> A quick fix for this could be just choosing either VS configuration
>> >>> >> when configuring the CCDC. Alternatively, HS_VS interrupts could be
>> >>> >> just disabled until omap3isp_configure_interface().
>> >>> >>
>> >>> >> But as the sensor is sending images all the time, proper VS
>> >>> >> configuration would be needed, or the counting of lines in the CCDC
>> >>> >> (VD* interrupts) is affected as well. The VD0 interrupt, which is
>> >>> >> used to trigger an interrupt near the end of the frame, may be
>> >>> >> triggered one line too early on the first frame, or too late. But
>> >>> >> this is up to a configuration. I don't think it's a real issue to
>> >>> >> trigger it one line too early.
>> >>> >>
>> >>> >> Anything else?
>> >>>
>> >>> Hello Laurent,
>> >>>
>> >>> > I've tried delaying the HS_VS interrupt enable to the CCDC
>> >>> > configuration function, after configuring the bridge (and thus the
>> >>> > HS/VS interrupt source selection). To my surprise it didn't fix the
>> >>> > problem, I still get tons of HS_VS interrupts (100000 in about 2.6
>> >>> > seconds) that kill the system.
>> >>> >
>> >>> > I'll need to hook a scope to the HS and VS signals.
>> >>>
>> >>> have you worked on this problem? Today in my setup I took a longer
>> >>> cable and ran again into the hs/vs interrupt storm (it still works
>> >>> with a short cable).
>> >>> I can tackle this issue too, but to avoid double work I wanted to ask
>> >>> if you worked out something in the meantime.
>> >>
>> >> In my case the issue was caused by a combination of two hardware design
>> >> mistakes. The first one was to use a TXB0801 chip to translate the 3.3V
>> >> sensor levels to the 1.8V OMAP levels. The TXB0801 4kΩ output
>> >> impedance, combined with the OMAP3 100µA pull-ups on the HS and VS
>> >> signals, produces a ~400mV voltage for low logic levels.
>> >>
>> >> Then, the XCLKA signal is next to the VS signal on the cable connecting
>> >> the camera module to the OMAP board. When XCLKA is turned on,
>> >> cross-talk produces a 400mV peak-to-peak noise on the VS signal.
>> >>
>> >> The combination of those two effects create a noisy VS signal that
>> >> crosses the OMAP3 input level detection gap at high frequency, leading
>> >> to an interrupt storm. The workaround is to disable the pull-ups on the
>> >> HS and VS signals, the solution is to redesign the hardware to replace
>> >> the level translators and reorganize signals on the camera module
>> >> cable.
>> >
>> > Hi Laurent,
>> >
>> >> Is your situation any similar ?
>> >
>> > The long data line (~35cm now at 24MHz) certainly can have an impact
>> > but I haven't measured any crosstalk so far. But I'm on another trail
>> > now. I found out that on my board the interrupt line is shared with
>> >  24:          0        INTC  omap-iommu.0
>> >
>> > Is the following scenario possible?
>> >
>> > 1. The omap-iommu isr is registered
>> > 2. The isp gets set up (it enables interrupts and disables them again
>> > at the end of the probe function)
>> > 3. Later I activate the xclk from within my driver
>> >  3a. isp_set_xclk() gets the lock omap3isp_get(isp) and so
>> > enable_interrupts() is called
>> >  3b. The new xclk on my chip makes my hardware create a hs/vs int
>> > (either crosstalk, another hardware bug like yours, or simply my chip
>> > sends a spurious interrupt for any reason)
>> >  3c.  isp_set_xclk() puts the lock omap3isp_put(isp) and so
>> > disable_interrupts() is called
>> >
>> > Can there exist a race condition between the omap3isp raising the
>> > interrupt pin before 3c or after 3c?
>>
>> Argh... I oversaw that the omap3isp isr handler stays registered all
>> time long so the theory is wrong.
>
> No luck :-)
>
> The first investigation step is to check which interrupt source causes the
> interrupts storm. The following test patch should help.
>
> diff --git a/drivers/media/video/omap3isp/isp.c b/drivers/media/video/omap3isp/isp.c
> index de2dec5..c4e6455 100644
> --- a/drivers/media/video/omap3isp/isp.c
> +++ b/drivers/media/video/omap3isp/isp.c
> @@ -400,6 +400,38 @@ static inline void isp_isr_dbg(struct isp_device *isp, u32 irqstatus)
>        printk(KERN_CONT "\n");
>  }
>
> +static unsigned int isp_isr_count[32];
> +
> +static inline void isp_isr_account(struct isp_device *isp, u32 irqstatus)
> +{
> +       unsigned int i;
> +
> +       spin_lock(&isp->isr_account_lock);
> +       for (i = 0; i < 32; i++) {
> +               if (irqstatus & (1 << i))
> +                       isp_isr_count[i]++;
> +       }
> +       spin_unlock(&isp->isr_account_lock);
> +}
> +
> +static ssize_t isp_isr_account_show(struct device *dev,
> +       struct device_attribute *attr, char *buf)
> +{
> +       struct isp_device *isp = container_of(to_media_device(to_media_devnode(dev)), struct isp_device, media_dev);
> +       unsigned long flags;
> +       unsigned int i;
> +       int ret = 0;
> +
> +       spin_lock_irqsave(&isp->isr_account_lock, flags);
> +       for (i = 0; i < 32; i++)
> +               ret += sprintf(buf + ret, "%u\t%u\n", i, isp_isr_count[i]);
> +       spin_unlock_irqrestore(&isp->isr_account_lock, flags);
> +
> +       return ret;
> +}
> +
> +static DEVICE_ATTR(isr_account, S_IRUGO, isp_isr_account_show, NULL);
> +
>  static void isp_isr_sbl(struct isp_device *isp)
>  {
>        struct device *dev = isp->dev;
> @@ -462,6 +494,7 @@ static irqreturn_t isp_isr(int irq, void *_isp)
>                                       IRQ0STATUS_CCDC_VD0_IRQ |
>                                       IRQ0STATUS_CCDC_VD1_IRQ |
>                                       IRQ0STATUS_HS_VS_IRQ;
> +       static unsigned int count = 0;
>        struct isp_device *isp = _isp;
>        u32 irqstatus;
>        int ret;
> @@ -469,6 +502,10 @@ static irqreturn_t isp_isr(int irq, void *_isp)
>        irqstatus = isp_reg_readl(isp, OMAP3_ISP_IOMEM_MAIN, ISP_IRQ0STATUS);
>        isp_reg_writel(isp, irqstatus, OMAP3_ISP_IOMEM_MAIN, ISP_IRQ0STATUS);
>
> +       isp_isr_account(isp, irqstatus);
> +       if (count++ >= 100000)
> +               isp_disable_interrupts(isp);
> +
>        isp_isr_sbl(isp);
>
>        if (irqstatus & IRQ0STATUS_CSIA_IRQ) {
> @@ -1971,6 +2008,7 @@ static int isp_remove(struct platform_device *pdev)
>        struct isp_device *isp = platform_get_drvdata(pdev);
>        int i;
>
> +       device_remove_file(&isp->media_dev.devnode.dev, &dev_attr_isr_account);
>        isp_unregister_entities(isp);
>        isp_cleanup_modules(isp);
>
> @@ -2067,6 +2105,7 @@ static int isp_probe(struct platform_device *pdev)
>
>        mutex_init(&isp->isp_mutex);
>        spin_lock_init(&isp->stat_lock);
> +       spin_lock_init(&isp->isr_account_lock);
>
>        isp->dev = &pdev->dev;
>        isp->pdata = pdata;
> @@ -2156,6 +2195,8 @@ static int isp_probe(struct platform_device *pdev)
>        isp_power_settings(isp, 1);
>        omap3isp_put(isp);
>
> +       ret = device_create_file(&isp->media_dev.devnode.dev, &dev_attr_isr_account);
> +
>        return 0;
>
>  error_modules:
> diff --git a/drivers/media/video/omap3isp/isp.h b/drivers/media/video/omap3isp/isp.h
> index 2620c40..b3f8448 100644
> --- a/drivers/media/video/omap3isp/isp.h
> +++ b/drivers/media/video/omap3isp/isp.h
> @@ -259,6 +259,7 @@ struct isp_device {
>
>        /* ISP Obj */
>        spinlock_t stat_lock;   /* common lock for statistic drivers */
> +       spinlock_t isr_account_lock;
>        struct mutex isp_mutex; /* For handling ref_count field */
>        bool needs_reset;
>        int has_context;
>
> Start a capture, wait a couple of settings for ISP interrupts to get disabled,
> and cat the isr_account sysfs file
> (/sys/bus/platform/devices/omap3isp/media0/isr_account if I remember
> correctly). My guess is that you will get approximatively 100000 HS_VS
> interrupts (31). Let's then move on from there after you've confirmed that the
> guess is correct.

Hello Laurent,

thank you very much for the patch. It are indeed hs/vs interrupts.

I discovered a heisenbug in my setup :)

When the igep stuck in the interrupts I meassured the hs line. I saw a
slight offset from ground. As I wondered how this small offset can
trigger the interrupt I realized that my system was running again. So
debugging my system with the oscilloscope made it run again. I try to
terminate the hs line with a pulldown to ground and see if I can
"simulate the debugging".
I probably had 24MI/s (megainterrupts per second) cause hs copied the
xclk wave (I discovered it before applying the patch)

Best regards,

 Bastian


> --
> Regards,
>
> Laurent Pinchart
>

Re: OMAP3 ISP deadlocks on my new arm

[Laurent Pinchart]

Hi Bastian,

On Thursday 28 April 2011 13:04:15 Bastian Hecht wrote:
> 2011/4/27 Laurent Pinchart <laurent.pinchart@ideasonboard.com>:
> > On Wednesday 27 April 2011 12:55:24 Bastian Hecht wrote:
> >> 2011/4/27 Bastian Hecht <hechtb@googlemail.com>:
> >> > 2011/4/26 Laurent Pinchart <laurent.pinchart@ideasonboard.com>:
> >> >> On Tuesday 26 April 2011 17:39:41 Bastian Hecht wrote:
> >> >>> 2011/4/21 Laurent Pinchart <laurent.pinchart@ideasonboard.com>:
> >> >>> > On Tuesday 19 April 2011 09:31:05 Sakari Ailus wrote:
> >> >>> >> Laurent Pinchart wrote:
> >> >>> >> ...
> >> >>> >> 
> >> >>> >> > That's the ideal situation: sensors should not produce any data
> >> >>> >> > (or rather any transition on the VS/HS signals) when they're
> >> >>> >> > supposed to be stopped. Unfortunately that's not always easy,
> >> >>> >> > as some dumb sensors (or sensor-like hardware) can't be
> >> >>> >> > stopped. The ISP driver should be able to cope with that in a
> >> >>> >> > way that doesn't kill the system completely.
> >> >>> >> > 
> >> >>> >> > I've noticed the same issue with a Caspa camera module and an
> >> >>> >> > OMAP3503-based Gumstix. I'll try to come up with a good fix.
> >> >>> >> 
> >> >>> >> Hi Laurent, others,
> >> >>> >> 
> >> >>> >> Do you think the cause for this is that the system is jammed in
> >> >>> >> handling HS_VS interrupts triggered for every HS?
> >> >>> > 
> >> >>> > That was my initial guess, yes.
> >> >>> > 
> >> >>> >> A quick fix for this could be just choosing either VS
> >> >>> >> configuration when configuring the CCDC. Alternatively, HS_VS
> >> >>> >> interrupts could be just disabled until
> >> >>> >> omap3isp_configure_interface().
> >> >>> >> 
> >> >>> >> But as the sensor is sending images all the time, proper VS
> >> >>> >> configuration would be needed, or the counting of lines in the
> >> >>> >> CCDC (VD* interrupts) is affected as well. The VD0 interrupt,
> >> >>> >> which is used to trigger an interrupt near the end of the frame,
> >> >>> >> may be triggered one line too early on the first frame, or too
> >> >>> >> late. But this is up to a configuration. I don't think it's a
> >> >>> >> real issue to trigger it one line too early.
> >> >>> >> 
> >> >>> >> Anything else?
> >> >>> 
> >> >>> Hello Laurent,
> >> >>> 
> >> >>> > I've tried delaying the HS_VS interrupt enable to the CCDC
> >> >>> > configuration function, after configuring the bridge (and thus the
> >> >>> > HS/VS interrupt source selection). To my surprise it didn't fix
> >> >>> > the problem, I still get tons of HS_VS interrupts (100000 in
> >> >>> > about 2.6 seconds) that kill the system.
> >> >>> > 
> >> >>> > I'll need to hook a scope to the HS and VS signals.
> >> >>> 
> >> >>> have you worked on this problem? Today in my setup I took a longer
> >> >>> cable and ran again into the hs/vs interrupt storm (it still works
> >> >>> with a short cable).
> >> >>> I can tackle this issue too, but to avoid double work I wanted to
> >> >>> ask if you worked out something in the meantime.
> >> >> 
> >> >> In my case the issue was caused by a combination of two hardware
> >> >> design mistakes. The first one was to use a TXB0801 chip to
> >> >> translate the 3.3V sensor levels to the 1.8V OMAP levels. The
> >> >> TXB0801 4kΩ output impedance, combined with the OMAP3 100µA pull-ups
> >> >> on the HS and VS signals, produces a ~400mV voltage for low logic
> >> >> levels.
> >> >> 
> >> >> Then, the XCLKA signal is next to the VS signal on the cable
> >> >> connecting the camera module to the OMAP board. When XCLKA is turned
> >> >> on, cross-talk produces a 400mV peak-to-peak noise on the VS signal.
> >> >> 
> >> >> The combination of those two effects create a noisy VS signal that
> >> >> crosses the OMAP3 input level detection gap at high frequency,
> >> >> leading to an interrupt storm. The workaround is to disable the
> >> >> pull-ups on the HS and VS signals, the solution is to redesign the
> >> >> hardware to replace the level translators and reorganize signals on
> >> >> the camera module cable.
> >> > 
> >> > Hi Laurent,
> >> > 
> >> >> Is your situation any similar ?
> >> > 
> >> > The long data line (~35cm now at 24MHz) certainly can have an impact
> >> > but I haven't measured any crosstalk so far. But I'm on another trail
> >> > now. I found out that on my board the interrupt line is shared with
> >> >  24:          0        INTC  omap-iommu.0
> >> > 
> >> > Is the following scenario possible?
> >> > 
> >> > 1. The omap-iommu isr is registered
> >> > 2. The isp gets set up (it enables interrupts and disables them again
> >> > at the end of the probe function)
> >> > 3. Later I activate the xclk from within my driver
> >> >  3a. isp_set_xclk() gets the lock omap3isp_get(isp) and so
> >> > enable_interrupts() is called
> >> >  3b. The new xclk on my chip makes my hardware create a hs/vs int
> >> > (either crosstalk, another hardware bug like yours, or simply my chip
> >> > sends a spurious interrupt for any reason)
> >> >  3c.  isp_set_xclk() puts the lock omap3isp_put(isp) and so
> >> > disable_interrupts() is called
> >> > 
> >> > Can there exist a race condition between the omap3isp raising the
> >> > interrupt pin before 3c or after 3c?
> >> 
> >> Argh... I oversaw that the omap3isp isr handler stays registered all
> >> time long so the theory is wrong.
> > 
> > No luck :-)
> > 
> > The first investigation step is to check which interrupt source causes
> > the interrupts storm. The following test patch should help.
> > 
> > diff --git a/drivers/media/video/omap3isp/isp.c
> > b/drivers/media/video/omap3isp/isp.c index de2dec5..c4e6455 100644
> > --- a/drivers/media/video/omap3isp/isp.c
> > +++ b/drivers/media/video/omap3isp/isp.c
> > @@ -400,6 +400,38 @@ static inline void isp_isr_dbg(struct isp_device
> > *isp, u32 irqstatus) printk(KERN_CONT "\n");
> >  }
> > 
> > +static unsigned int isp_isr_count[32];
> > +
> > +static inline void isp_isr_account(struct isp_device *isp, u32
> > irqstatus) +{
> > +       unsigned int i;
> > +
> > +       spin_lock(&isp->isr_account_lock);
> > +       for (i = 0; i < 32; i++) {
> > +               if (irqstatus & (1 << i))
> > +                       isp_isr_count[i]++;
> > +       }
> > +       spin_unlock(&isp->isr_account_lock);
> > +}
> > +
> > +static ssize_t isp_isr_account_show(struct device *dev,
> > +       struct device_attribute *attr, char *buf)
> > +{
> > +       struct isp_device *isp =
> > container_of(to_media_device(to_media_devnode(dev)), struct isp_device,
> > media_dev); +       unsigned long flags;
> > +       unsigned int i;
> > +       int ret = 0;
> > +
> > +       spin_lock_irqsave(&isp->isr_account_lock, flags);
> > +       for (i = 0; i < 32; i++)
> > +               ret += sprintf(buf + ret, "%u\t%u\n", i,
> > isp_isr_count[i]); +      
> > spin_unlock_irqrestore(&isp->isr_account_lock, flags);
> > +
> > +       return ret;
> > +}
> > +
> > +static DEVICE_ATTR(isr_account, S_IRUGO, isp_isr_account_show, NULL);
> > +
> >  static void isp_isr_sbl(struct isp_device *isp)
> >  {
> >        struct device *dev = isp->dev;
> > @@ -462,6 +494,7 @@ static irqreturn_t isp_isr(int irq, void *_isp)
> >                                       IRQ0STATUS_CCDC_VD0_IRQ |
> >                                       IRQ0STATUS_CCDC_VD1_IRQ |
> >                                       IRQ0STATUS_HS_VS_IRQ;
> > +       static unsigned int count = 0;
> >        struct isp_device *isp = _isp;
> >        u32 irqstatus;
> >        int ret;
> > @@ -469,6 +502,10 @@ static irqreturn_t isp_isr(int irq, void *_isp)
> >        irqstatus = isp_reg_readl(isp, OMAP3_ISP_IOMEM_MAIN,
> > ISP_IRQ0STATUS); isp_reg_writel(isp, irqstatus, OMAP3_ISP_IOMEM_MAIN,
> > ISP_IRQ0STATUS);
> > 
> > +       isp_isr_account(isp, irqstatus);
> > +       if (count++ >= 100000)
> > +               isp_disable_interrupts(isp);
> > +
> >        isp_isr_sbl(isp);
> > 
> >        if (irqstatus & IRQ0STATUS_CSIA_IRQ) {
> > @@ -1971,6 +2008,7 @@ static int isp_remove(struct platform_device *pdev)
> >        struct isp_device *isp = platform_get_drvdata(pdev);
> >        int i;
> > 
> > +       device_remove_file(&isp->media_dev.devnode.dev,
> > &dev_attr_isr_account); isp_unregister_entities(isp);
> >        isp_cleanup_modules(isp);
> > 
> > @@ -2067,6 +2105,7 @@ static int isp_probe(struct platform_device *pdev)
> > 
> >        mutex_init(&isp->isp_mutex);
> >        spin_lock_init(&isp->stat_lock);
> > +       spin_lock_init(&isp->isr_account_lock);
> > 
> >        isp->dev = &pdev->dev;
> >        isp->pdata = pdata;
> > @@ -2156,6 +2195,8 @@ static int isp_probe(struct platform_device *pdev)
> >        isp_power_settings(isp, 1);
> >        omap3isp_put(isp);
> > 
> > +       ret = device_create_file(&isp->media_dev.devnode.dev,
> > &dev_attr_isr_account); +
> >        return 0;
> > 
> >  error_modules:
> > diff --git a/drivers/media/video/omap3isp/isp.h
> > b/drivers/media/video/omap3isp/isp.h index 2620c40..b3f8448 100644
> > --- a/drivers/media/video/omap3isp/isp.h
> > +++ b/drivers/media/video/omap3isp/isp.h
> > @@ -259,6 +259,7 @@ struct isp_device {
> > 
> >        /* ISP Obj */
> >        spinlock_t stat_lock;   /* common lock for statistic drivers */
> > +       spinlock_t isr_account_lock;
> >        struct mutex isp_mutex; /* For handling ref_count field */
> >        bool needs_reset;
> >        int has_context;
> > 
> > Start a capture, wait a couple of settings for ISP interrupts to get
> > disabled, and cat the isr_account sysfs file
> > (/sys/bus/platform/devices/omap3isp/media0/isr_account if I remember
> > correctly). My guess is that you will get approximatively 100000 HS_VS
> > interrupts (31). Let's then move on from there after you've confirmed
> > that the guess is correct.
> 
> Hello Laurent,
> 
> thank you very much for the patch. It are indeed hs/vs interrupts.

You're welcome.

> I discovered a heisenbug in my setup :)
> 
> When the igep stuck in the interrupts I meassured the hs line. I saw a
> slight offset from ground. As I wondered how this small offset can
> trigger the interrupt I realized that my system was running again. So
> debugging my system with the oscilloscope made it run again. I try to
> terminate the hs line with a pulldown to ground and see if I can
> "simulate the debugging".
> I probably had 24MI/s (megainterrupts per second) cause hs copied the
> xclk wave (I discovered it before applying the patch)

That's way too much interrupts :-)

So your problem is now solved with the pulldown ? Instead of modifying the 
hardware, you can enable the OMAP3 internal pull-up or pull-down on the HS and 
VS pins if they haven't been enabled by the boot loader by applying the 
following patch to your board file.

 #ifdef CONFIG_OMAP_MUX
 static struct omap_board_mux board_mux[] __initdata = {
+       /* CAM */
+       OMAP3_MUX(CAM_HS, OMAP_MUX_MODE0 | OMAP_PIN_INPUT_PULLUP),
+       OMAP3_MUX(CAM_VS, OMAP_MUX_MODE0 | OMAP_PIN_INPUT_PULLUP),
+
        { .reg_offset = OMAP_MUX_TERMINATOR },
 };
 #endif

-- 
Regards,

Laurent Pinchart