* [BUG] 2.6.26-rc1-git4 - task blocked on powerpc for more than 120 seconds @ 2008-05-07 13:22 Kamalesh Babulal 2008-05-07 13:40 ` Adrian Bunk 0 siblings, 1 reply; 8+ messages in thread From: Kamalesh Babulal @ 2008-05-07 13:22 UTC (permalink / raw) To: kernel list; +Cc: linuxppc-dev, Balbir Singh While running the ltp over the powerpc with the 2.6.26-rc1-git4 kernel, task get blocked for more 120 seconds and does not proceeds future. The task msgctl08 is a ipc testcase, which test the msgget(2) msgctl(2) syscalls. Machine is stuck in the task, printing the following call trace more than 5000 times. The oom-killer invoked once in-between. INFO: task msgctl08:16248 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Call Trace: [c0000000762435a0] [0000000000000001] 0x1 (unreliable) [c000000076243770] [c000000000010acc] .__switch_to+0x128/0x16c [c000000076243800] [c0000000004b07a4] .schedule+0x7ac/0x890 [c0000000762438f0] [c0000000004b2ba4] .rwsem_down_failed_common+0x260/0x2b0 [c0000000762439b0] [c0000000004b2c60] .rwsem_down_read_failed+0x2c/0x44 [c000000076243a60] [c0000000004b1b84] .down_read+0x44/0x5c [c000000076243af0] [c000000000295e10] .ipc_lock+0x34/0xe0 [c000000076243b90] [c00000000029690c] .ipc_lock_check+0x24/0x78 [c000000076243c20] [c0000000002972c0] .do_msgsnd+0xb0/0x3f8 [c000000076243d10] [c000000000293ce8] .compat_sys_msgsnd+0x94/0xc0 [c000000076243db0] [c000000000014418] .compat_sys_ipc+0x130/0x1f4 [c000000076243e30] [c000000000008734] syscall_exit+0x0/0x40 msgctl08 invoked oom-killer: gfp_mask=0x1200d2, order=0, oomkilladj=0 Call Trace: [c00000001e1c7640] [c0000000000101bc] .show_stack+0x70/0x1bc (unreliable) [c00000001e1c76f0] [c0000000000c2c78] .oom_kill_process+0x78/0x230 [c00000001e1c77a0] [c0000000000c3390] .out_of_memory+0x28c/0x320 [c00000001e1c7870] [c0000000000c70ac] .__alloc_pages_internal+0x364/0x468 [c00000001e1c7980] [c0000000000e83cc] .alloc_page_vma+0x120/0x14c [c00000001e1c7a20] [c0000000000e0224] .read_swap_cache_async+0x7c/0x160 [c00000001e1c7ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 [c00000001e1c7ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 [c00000001e1c7c80] [c0000000004b5054] .do_page_fault+0x440/0x624 [c00000001e1c7e30] [c0000000000053fc] handle_page_fault+0x20/0x5c Mem-info: Node 0 DMA per-cpu: CPU 0: hi: 6, btch: 1 usd: 1 CPU 1: hi: 6, btch: 1 usd: 1 Node 1 DMA per-cpu: CPU 0: hi: 6, btch: 1 usd: 5 CPU 1: hi: 6, btch: 1 usd: 5 Active:31 inactive:2628 dirty:1 writeback:6 unstable:0 free:156 slab:13071 mapped:548 pagetables:41109 bounce:0 Node 0 DMA free:5312kB min:5760kB low:7168kB high:8640kB active:1024kB inactive:768kB present:3928832kB pages_scanned:3912 all_unreclaimable? no lowmem_reserve[]: 0 0 0 Node 1 DMA free:4672kB min:4992kB low:6208kB high:7488kB active:960kB inactive:169792kB present:3404992kB pages_scanned:140140 all_unreclaimable? no lowmem_reserve[]: 0 0 0 Node 0 DMA: 5*64kB 15*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB 0*8192kB 0*16384kB = 5312kB Node 1 DMA: 8*64kB 5*128kB 6*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB 0*8192kB 0*16384kB = 4736kB 8185 total pagecache pages Swap cache: add 29545, delete 21888, find 5602/10373 Free swap = 803712kB Total swap = 2048128kB 114688 pages of RAM 766 reserved pages 231218 pages shared 7657 pages swap cached Out of memory: kill process 15371 (msgctl08) score 39223 or a child Killed process 15373 (msgctl08) INFO: task msgctl08:15385 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Call Trace: [c0000000e9a43290] [0000000000000001] 0x1 (unreliable) [c0000000e9a43460] [c000000000010acc] .__switch_to+0x128/0x16c [c0000000e9a434f0] [c0000000004b07a4] .schedule+0x7ac/0x890 [c0000000e9a435e0] [c0000000004b0d90] .io_schedule+0x7c/0xe8 [c0000000e9a43670] [c0000000002d7f44] .get_request_wait+0x15c/0x1e0 [c0000000e9a43750] [c0000000002d8950] .__make_request+0x3ec/0x4d8 [c0000000e9a43800] [c0000000002d6624] .generic_make_request+0x35c/0x3b0 [c0000000e9a438e0] [c0000000002d81b0] .submit_bio+0x118/0x140 [c0000000e9a439a0] [c0000000000dfa88] .swap_readpage+0x94/0xb4 [c0000000e9a43a20] [c0000000000e02b8] .read_swap_cache_async+0x110/0x160 [c0000000e9a43ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 [c0000000e9a43ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 [c0000000e9a43c80] [c0000000004b5054] .do_page_fault+0x440/0x624 [c0000000e9a43e30] [c0000000000053fc] handle_page_fault+0x20/0x5c INFO: task msgctl08:15405 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Call Trace: [c0000000b1757290] [0000000000000001] 0x1 (unreliable) [c0000000b1757460] [c000000000010acc] .__switch_to+0x128/0x16c [c0000000b17574f0] [c0000000004b07a4] .schedule+0x7ac/0x890 [c0000000b17575e0] [c0000000004b0d90] .io_schedule+0x7c/0xe8 [c0000000b1757670] [c0000000002d7f44] .get_request_wait+0x15c/0x1e0 [c0000000b1757750] [c0000000002d8950] .__make_request+0x3ec/0x4d8 [c0000000b1757800] [c0000000002d6624] .generic_make_request+0x35c/0x3b0 [c0000000b17578e0] [c0000000002d81b0] .submit_bio+0x118/0x140 [c0000000b17579a0] [c0000000000dfa88] .swap_readpage+0x94/0xb4 [c0000000b1757a20] [c0000000000e02b8] .read_swap_cache_async+0x110/0x160 [c0000000b1757ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 [c0000000b1757ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 [c0000000b1757c80] [c0000000004b5054] .do_page_fault+0x440/0x624 [c0000000b1757e30] [c0000000000053fc] handle_page_fault+0x20/0x5c INFO: task msgctl08:15406 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Call Trace: [c0000000eec53290] [0000000000000001] 0x1 (unreliable) [c0000000eec53460] [c000000000010acc] .__switch_to+0x128/0x16c [c0000000eec534f0] [c0000000004b07a4] .schedule+0x7ac/0x890 [c0000000eec535e0] [c0000000004b0d90] .io_schedule+0x7c/0xe8 [c0000000eec53670] [c0000000002d7f44] .get_request_wait+0x15c/0x1e0 [c0000000eec53750] [c0000000002d8950] .__make_request+0x3ec/0x4d8 [c0000000eec53800] [c0000000002d6624] .generic_make_request+0x35c/0x3b0 [c0000000eec538e0] [c0000000002d81b0] .submit_bio+0x118/0x140 [c0000000eec539a0] [c0000000000dfa88] .swap_readpage+0x94/0xb4 [c0000000eec53a20] [c0000000000e02b8] .read_swap_cache_async+0x110/0x160 [c0000000eec53ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 [c0000000eec53ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 [c0000000eec53c80] [c0000000004b5054] .do_page_fault+0x440/0x624 [c0000000eec53e30] [c0000000000053fc] handle_page_fault+0x20/0x5c -- Thanks & Regards, Kamalesh Babulal, Linux Technology Center, IBM, ISTL. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [BUG] 2.6.26-rc1-git4 - task blocked on powerpc for more than 120 seconds 2008-05-07 13:22 [BUG] 2.6.26-rc1-git4 - task blocked on powerpc for more than 120 seconds Kamalesh Babulal @ 2008-05-07 13:40 ` Adrian Bunk 2008-05-08 3:59 ` Kamalesh Babulal 0 siblings, 1 reply; 8+ messages in thread From: Adrian Bunk @ 2008-05-07 13:40 UTC (permalink / raw) To: Kamalesh Babulal Cc: kernel list, linuxppc-dev, Pierre Peiffer, Nadia Derbey, Balbir Singh On Wed, May 07, 2008 at 06:52:53PM +0530, Kamalesh Babulal wrote: > While running the ltp over the powerpc with the 2.6.26-rc1-git4 kernel, > task get blocked for more 120 seconds and does not proceeds future. > > The task msgctl08 is a ipc testcase, which test the msgget(2) msgctl(2) > syscalls. Thanks for your report. I assume this is reproducible? If yes, what was the last kernel that worked for you? I've also added Pierre Peiffer and Nadia Derbey to the Cc since their recent ipc commits are my first suspects. > Machine is stuck in the task, printing the following call trace > more than 5000 times. The oom-killer invoked once in-between. > > INFO: task msgctl08:16248 blocked for more than 120 seconds. > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > Call Trace: > [c0000000762435a0] [0000000000000001] 0x1 (unreliable) > [c000000076243770] [c000000000010acc] .__switch_to+0x128/0x16c > [c000000076243800] [c0000000004b07a4] .schedule+0x7ac/0x890 > [c0000000762438f0] [c0000000004b2ba4] .rwsem_down_failed_common+0x260/0x2b0 > [c0000000762439b0] [c0000000004b2c60] .rwsem_down_read_failed+0x2c/0x44 > [c000000076243a60] [c0000000004b1b84] .down_read+0x44/0x5c > [c000000076243af0] [c000000000295e10] .ipc_lock+0x34/0xe0 > [c000000076243b90] [c00000000029690c] .ipc_lock_check+0x24/0x78 > [c000000076243c20] [c0000000002972c0] .do_msgsnd+0xb0/0x3f8 > [c000000076243d10] [c000000000293ce8] .compat_sys_msgsnd+0x94/0xc0 > [c000000076243db0] [c000000000014418] .compat_sys_ipc+0x130/0x1f4 > [c000000076243e30] [c000000000008734] syscall_exit+0x0/0x40 > msgctl08 invoked oom-killer: gfp_mask=0x1200d2, order=0, oomkilladj=0 > Call Trace: > [c00000001e1c7640] [c0000000000101bc] .show_stack+0x70/0x1bc (unreliable) > [c00000001e1c76f0] [c0000000000c2c78] .oom_kill_process+0x78/0x230 > [c00000001e1c77a0] [c0000000000c3390] .out_of_memory+0x28c/0x320 > [c00000001e1c7870] [c0000000000c70ac] .__alloc_pages_internal+0x364/0x468 > [c00000001e1c7980] [c0000000000e83cc] .alloc_page_vma+0x120/0x14c > [c00000001e1c7a20] [c0000000000e0224] .read_swap_cache_async+0x7c/0x160 > [c00000001e1c7ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 > [c00000001e1c7ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 > [c00000001e1c7c80] [c0000000004b5054] .do_page_fault+0x440/0x624 > [c00000001e1c7e30] [c0000000000053fc] handle_page_fault+0x20/0x5c > Mem-info: > Node 0 DMA per-cpu: > CPU 0: hi: 6, btch: 1 usd: 1 > CPU 1: hi: 6, btch: 1 usd: 1 > Node 1 DMA per-cpu: > CPU 0: hi: 6, btch: 1 usd: 5 > CPU 1: hi: 6, btch: 1 usd: 5 > Active:31 inactive:2628 dirty:1 writeback:6 unstable:0 > free:156 slab:13071 mapped:548 pagetables:41109 bounce:0 > Node 0 DMA free:5312kB min:5760kB low:7168kB high:8640kB active:1024kB inactive:768kB present:3928832kB pages_scanned:3912 all_unreclaimable? no > lowmem_reserve[]: 0 0 0 > Node 1 DMA free:4672kB min:4992kB low:6208kB high:7488kB active:960kB inactive:169792kB present:3404992kB pages_scanned:140140 all_unreclaimable? no > lowmem_reserve[]: 0 0 0 > Node 0 DMA: 5*64kB 15*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB 0*8192kB 0*16384kB = 5312kB > Node 1 DMA: 8*64kB 5*128kB 6*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB 0*8192kB 0*16384kB = 4736kB > 8185 total pagecache pages > Swap cache: add 29545, delete 21888, find 5602/10373 > Free swap = 803712kB > Total swap = 2048128kB > 114688 pages of RAM > 766 reserved pages > 231218 pages shared > 7657 pages swap cached > Out of memory: kill process 15371 (msgctl08) score 39223 or a child > Killed process 15373 (msgctl08) > INFO: task msgctl08:15385 blocked for more than 120 seconds. > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > Call Trace: > [c0000000e9a43290] [0000000000000001] 0x1 (unreliable) > [c0000000e9a43460] [c000000000010acc] .__switch_to+0x128/0x16c > [c0000000e9a434f0] [c0000000004b07a4] .schedule+0x7ac/0x890 > [c0000000e9a435e0] [c0000000004b0d90] .io_schedule+0x7c/0xe8 > [c0000000e9a43670] [c0000000002d7f44] .get_request_wait+0x15c/0x1e0 > [c0000000e9a43750] [c0000000002d8950] .__make_request+0x3ec/0x4d8 > [c0000000e9a43800] [c0000000002d6624] .generic_make_request+0x35c/0x3b0 > [c0000000e9a438e0] [c0000000002d81b0] .submit_bio+0x118/0x140 > [c0000000e9a439a0] [c0000000000dfa88] .swap_readpage+0x94/0xb4 > [c0000000e9a43a20] [c0000000000e02b8] .read_swap_cache_async+0x110/0x160 > [c0000000e9a43ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 > [c0000000e9a43ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 > [c0000000e9a43c80] [c0000000004b5054] .do_page_fault+0x440/0x624 > [c0000000e9a43e30] [c0000000000053fc] handle_page_fault+0x20/0x5c > INFO: task msgctl08:15405 blocked for more than 120 seconds. > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > Call Trace: > [c0000000b1757290] [0000000000000001] 0x1 (unreliable) > [c0000000b1757460] [c000000000010acc] .__switch_to+0x128/0x16c > [c0000000b17574f0] [c0000000004b07a4] .schedule+0x7ac/0x890 > [c0000000b17575e0] [c0000000004b0d90] .io_schedule+0x7c/0xe8 > [c0000000b1757670] [c0000000002d7f44] .get_request_wait+0x15c/0x1e0 > [c0000000b1757750] [c0000000002d8950] .__make_request+0x3ec/0x4d8 > [c0000000b1757800] [c0000000002d6624] .generic_make_request+0x35c/0x3b0 > [c0000000b17578e0] [c0000000002d81b0] .submit_bio+0x118/0x140 > [c0000000b17579a0] [c0000000000dfa88] .swap_readpage+0x94/0xb4 > [c0000000b1757a20] [c0000000000e02b8] .read_swap_cache_async+0x110/0x160 > [c0000000b1757ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 > [c0000000b1757ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 > [c0000000b1757c80] [c0000000004b5054] .do_page_fault+0x440/0x624 > [c0000000b1757e30] [c0000000000053fc] handle_page_fault+0x20/0x5c > INFO: task msgctl08:15406 blocked for more than 120 seconds. > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > Call Trace: > [c0000000eec53290] [0000000000000001] 0x1 (unreliable) > [c0000000eec53460] [c000000000010acc] .__switch_to+0x128/0x16c > [c0000000eec534f0] [c0000000004b07a4] .schedule+0x7ac/0x890 > [c0000000eec535e0] [c0000000004b0d90] .io_schedule+0x7c/0xe8 > [c0000000eec53670] [c0000000002d7f44] .get_request_wait+0x15c/0x1e0 > [c0000000eec53750] [c0000000002d8950] .__make_request+0x3ec/0x4d8 > [c0000000eec53800] [c0000000002d6624] .generic_make_request+0x35c/0x3b0 > [c0000000eec538e0] [c0000000002d81b0] .submit_bio+0x118/0x140 > [c0000000eec539a0] [c0000000000dfa88] .swap_readpage+0x94/0xb4 > [c0000000eec53a20] [c0000000000e02b8] .read_swap_cache_async+0x110/0x160 > [c0000000eec53ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 > [c0000000eec53ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 > [c0000000eec53c80] [c0000000004b5054] .do_page_fault+0x440/0x624 > [c0000000eec53e30] [c0000000000053fc] handle_page_fault+0x20/0x5c > -- > Thanks & Regards, > Kamalesh Babulal, > Linux Technology Center, > IBM, ISTL. cu Adrian -- "Is there not promise of rain?" Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. "Only a promise," Lao Er said. Pearl S. Buck - Dragon Seed ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [BUG] 2.6.26-rc1-git4 - task blocked on powerpc for more than 120 seconds 2008-05-07 13:40 ` Adrian Bunk @ 2008-05-08 3:59 ` Kamalesh Babulal 2008-05-15 11:30 ` Nadia Derbey 0 siblings, 1 reply; 8+ messages in thread From: Kamalesh Babulal @ 2008-05-08 3:59 UTC (permalink / raw) To: Adrian Bunk Cc: kernel list, linuxppc-dev, Pierre Peiffer, Nadia Derbey, Balbir Singh Adrian Bunk wrote: > On Wed, May 07, 2008 at 06:52:53PM +0530, Kamalesh Babulal wrote: >> While running the ltp over the powerpc with the 2.6.26-rc1-git4 kernel, >> task get blocked for more 120 seconds and does not proceeds future. >> >> The task msgctl08 is a ipc testcase, which test the msgget(2) msgctl(2) >> syscalls. > > Thanks for your report. > > I assume this is reproducible? > > If yes, what was the last kernel that worked for you? This is reproducible on 2.6.26-rc1 and 2.6.26-rc1-git(s). 2.6.25 is the last know kernel to work. > > I've also added Pierre Peiffer and Nadia Derbey to the Cc since their > recent ipc commits are my first suspects. > >> Machine is stuck in the task, printing the following call trace >> more than 5000 times. The oom-killer invoked once in-between. >> >> INFO: task msgctl08:16248 blocked for more than 120 seconds. >> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. >> Call Trace: >> [c0000000762435a0] [0000000000000001] 0x1 (unreliable) >> [c000000076243770] [c000000000010acc] .__switch_to+0x128/0x16c >> [c000000076243800] [c0000000004b07a4] .schedule+0x7ac/0x890 >> [c0000000762438f0] [c0000000004b2ba4] .rwsem_down_failed_common+0x260/0x2b0 >> [c0000000762439b0] [c0000000004b2c60] .rwsem_down_read_failed+0x2c/0x44 >> [c000000076243a60] [c0000000004b1b84] .down_read+0x44/0x5c >> [c000000076243af0] [c000000000295e10] .ipc_lock+0x34/0xe0 >> [c000000076243b90] [c00000000029690c] .ipc_lock_check+0x24/0x78 >> [c000000076243c20] [c0000000002972c0] .do_msgsnd+0xb0/0x3f8 >> [c000000076243d10] [c000000000293ce8] .compat_sys_msgsnd+0x94/0xc0 >> [c000000076243db0] [c000000000014418] .compat_sys_ipc+0x130/0x1f4 >> [c000000076243e30] [c000000000008734] syscall_exit+0x0/0x40 >> msgctl08 invoked oom-killer: gfp_mask=0x1200d2, order=0, oomkilladj=0 >> Call Trace: >> [c00000001e1c7640] [c0000000000101bc] .show_stack+0x70/0x1bc (unreliable) >> [c00000001e1c76f0] [c0000000000c2c78] .oom_kill_process+0x78/0x230 >> [c00000001e1c77a0] [c0000000000c3390] .out_of_memory+0x28c/0x320 >> [c00000001e1c7870] [c0000000000c70ac] .__alloc_pages_internal+0x364/0x468 >> [c00000001e1c7980] [c0000000000e83cc] .alloc_page_vma+0x120/0x14c >> [c00000001e1c7a20] [c0000000000e0224] .read_swap_cache_async+0x7c/0x160 >> [c00000001e1c7ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 >> [c00000001e1c7ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 >> [c00000001e1c7c80] [c0000000004b5054] .do_page_fault+0x440/0x624 >> [c00000001e1c7e30] [c0000000000053fc] handle_page_fault+0x20/0x5c >> Mem-info: >> Node 0 DMA per-cpu: >> CPU 0: hi: 6, btch: 1 usd: 1 >> CPU 1: hi: 6, btch: 1 usd: 1 >> Node 1 DMA per-cpu: >> CPU 0: hi: 6, btch: 1 usd: 5 >> CPU 1: hi: 6, btch: 1 usd: 5 >> Active:31 inactive:2628 dirty:1 writeback:6 unstable:0 >> free:156 slab:13071 mapped:548 pagetables:41109 bounce:0 >> Node 0 DMA free:5312kB min:5760kB low:7168kB high:8640kB active:1024kB inactive:768kB present:3928832kB pages_scanned:3912 all_unreclaimable? no >> lowmem_reserve[]: 0 0 0 >> Node 1 DMA free:4672kB min:4992kB low:6208kB high:7488kB active:960kB inactive:169792kB present:3404992kB pages_scanned:140140 all_unreclaimable? no >> lowmem_reserve[]: 0 0 0 >> Node 0 DMA: 5*64kB 15*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB 0*8192kB 0*16384kB = 5312kB >> Node 1 DMA: 8*64kB 5*128kB 6*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB 0*8192kB 0*16384kB = 4736kB >> 8185 total pagecache pages >> Swap cache: add 29545, delete 21888, find 5602/10373 >> Free swap = 803712kB >> Total swap = 2048128kB >> 114688 pages of RAM >> 766 reserved pages >> 231218 pages shared >> 7657 pages swap cached >> Out of memory: kill process 15371 (msgctl08) score 39223 or a child >> Killed process 15373 (msgctl08) >> INFO: task msgctl08:15385 blocked for more than 120 seconds. >> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. >> Call Trace: >> [c0000000e9a43290] [0000000000000001] 0x1 (unreliable) >> [c0000000e9a43460] [c000000000010acc] .__switch_to+0x128/0x16c >> [c0000000e9a434f0] [c0000000004b07a4] .schedule+0x7ac/0x890 >> [c0000000e9a435e0] [c0000000004b0d90] .io_schedule+0x7c/0xe8 >> [c0000000e9a43670] [c0000000002d7f44] .get_request_wait+0x15c/0x1e0 >> [c0000000e9a43750] [c0000000002d8950] .__make_request+0x3ec/0x4d8 >> [c0000000e9a43800] [c0000000002d6624] .generic_make_request+0x35c/0x3b0 >> [c0000000e9a438e0] [c0000000002d81b0] .submit_bio+0x118/0x140 >> [c0000000e9a439a0] [c0000000000dfa88] .swap_readpage+0x94/0xb4 >> [c0000000e9a43a20] [c0000000000e02b8] .read_swap_cache_async+0x110/0x160 >> [c0000000e9a43ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 >> [c0000000e9a43ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 >> [c0000000e9a43c80] [c0000000004b5054] .do_page_fault+0x440/0x624 >> [c0000000e9a43e30] [c0000000000053fc] handle_page_fault+0x20/0x5c >> INFO: task msgctl08:15405 blocked for more than 120 seconds. >> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. >> Call Trace: >> [c0000000b1757290] [0000000000000001] 0x1 (unreliable) >> [c0000000b1757460] [c000000000010acc] .__switch_to+0x128/0x16c >> [c0000000b17574f0] [c0000000004b07a4] .schedule+0x7ac/0x890 >> [c0000000b17575e0] [c0000000004b0d90] .io_schedule+0x7c/0xe8 >> [c0000000b1757670] [c0000000002d7f44] .get_request_wait+0x15c/0x1e0 >> [c0000000b1757750] [c0000000002d8950] .__make_request+0x3ec/0x4d8 >> [c0000000b1757800] [c0000000002d6624] .generic_make_request+0x35c/0x3b0 >> [c0000000b17578e0] [c0000000002d81b0] .submit_bio+0x118/0x140 >> [c0000000b17579a0] [c0000000000dfa88] .swap_readpage+0x94/0xb4 >> [c0000000b1757a20] [c0000000000e02b8] .read_swap_cache_async+0x110/0x160 >> [c0000000b1757ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 >> [c0000000b1757ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 >> [c0000000b1757c80] [c0000000004b5054] .do_page_fault+0x440/0x624 >> [c0000000b1757e30] [c0000000000053fc] handle_page_fault+0x20/0x5c >> INFO: task msgctl08:15406 blocked for more than 120 seconds. >> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. >> Call Trace: >> [c0000000eec53290] [0000000000000001] 0x1 (unreliable) >> [c0000000eec53460] [c000000000010acc] .__switch_to+0x128/0x16c >> [c0000000eec534f0] [c0000000004b07a4] .schedule+0x7ac/0x890 >> [c0000000eec535e0] [c0000000004b0d90] .io_schedule+0x7c/0xe8 >> [c0000000eec53670] [c0000000002d7f44] .get_request_wait+0x15c/0x1e0 >> [c0000000eec53750] [c0000000002d8950] .__make_request+0x3ec/0x4d8 >> [c0000000eec53800] [c0000000002d6624] .generic_make_request+0x35c/0x3b0 >> [c0000000eec538e0] [c0000000002d81b0] .submit_bio+0x118/0x140 >> [c0000000eec539a0] [c0000000000dfa88] .swap_readpage+0x94/0xb4 >> [c0000000eec53a20] [c0000000000e02b8] .read_swap_cache_async+0x110/0x160 >> [c0000000eec53ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 >> [c0000000eec53ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 >> [c0000000eec53c80] [c0000000004b5054] .do_page_fault+0x440/0x624 >> [c0000000eec53e30] [c0000000000053fc] handle_page_fault+0x20/0x5c >> -- >> Thanks & Regards, >> Kamalesh Babulal, >> Linux Technology Center, >> IBM, ISTL. > > cu > Adrian -- Thanks & Regards, Kamalesh Babulal, Linux Technology Center, IBM, ISTL. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [BUG] 2.6.26-rc1-git4 - task blocked on powerpc for more than 120 seconds 2008-05-08 3:59 ` Kamalesh Babulal @ 2008-05-15 11:30 ` Nadia Derbey 2008-05-16 14:26 ` Kamalesh Babulal 0 siblings, 1 reply; 8+ messages in thread From: Nadia Derbey @ 2008-05-15 11:30 UTC (permalink / raw) To: Kamalesh Babulal; +Cc: linuxppc-dev, Adrian Bunk, Balbir Singh, kernel list Kamalesh Babulal wrote: > Adrian Bunk wrote: > >>On Wed, May 07, 2008 at 06:52:53PM +0530, Kamalesh Babulal wrote: >> >>>While running the ltp over the powerpc with the 2.6.26-rc1-git4 kernel, >>>task get blocked for more 120 seconds and does not proceeds future. >>> >>>The task msgctl08 is a ipc testcase, which test the msgget(2) msgctl(2) >>>syscalls. >> >>Thanks for your report. >> >>I assume this is reproducible? >> >>If yes, what was the last kernel that worked for you? > > > This is reproducible on 2.6.26-rc1 and 2.6.26-rc1-git(s). 2.6.25 is the last know kernel > to work. > > >>I've also added Pierre Peiffer and Nadia Derbey to the Cc since their >>recent ipc commits are my first suspects. >> >> >>>Machine is stuck in the task, printing the following call trace >>>more than 5000 times. The oom-killer invoked once in-between. >>> >>>INFO: task msgctl08:16248 blocked for more than 120 seconds. >>>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. >>>Call Trace: >>>[c0000000762435a0] [0000000000000001] 0x1 (unreliable) >>>[c000000076243770] [c000000000010acc] .__switch_to+0x128/0x16c >>>[c000000076243800] [c0000000004b07a4] .schedule+0x7ac/0x890 >>>[c0000000762438f0] [c0000000004b2ba4] .rwsem_down_failed_common+0x260/0x2b0 >>>[c0000000762439b0] [c0000000004b2c60] .rwsem_down_read_failed+0x2c/0x44 >>>[c000000076243a60] [c0000000004b1b84] .down_read+0x44/0x5c >>>[c000000076243af0] [c000000000295e10] .ipc_lock+0x34/0xe0 >>>[c000000076243b90] [c00000000029690c] .ipc_lock_check+0x24/0x78 >>>[c000000076243c20] [c0000000002972c0] .do_msgsnd+0xb0/0x3f8 >>>[c000000076243d10] [c000000000293ce8] .compat_sys_msgsnd+0x94/0xc0 >>>[c000000076243db0] [c000000000014418] .compat_sys_ipc+0x130/0x1f4 >>>[c000000076243e30] [c000000000008734] syscall_exit+0x0/0x40 >>>msgctl08 invoked oom-killer: gfp_mask=0x1200d2, order=0, oomkilladj=0 >>>Call Trace: >>>[c00000001e1c7640] [c0000000000101bc] .show_stack+0x70/0x1bc (unreliable) >>>[c00000001e1c76f0] [c0000000000c2c78] .oom_kill_process+0x78/0x230 >>>[c00000001e1c77a0] [c0000000000c3390] .out_of_memory+0x28c/0x320 >>>[c00000001e1c7870] [c0000000000c70ac] .__alloc_pages_internal+0x364/0x468 >>>[c00000001e1c7980] [c0000000000e83cc] .alloc_page_vma+0x120/0x14c >>>[c00000001e1c7a20] [c0000000000e0224] .read_swap_cache_async+0x7c/0x160 >>>[c00000001e1c7ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 >>>[c00000001e1c7ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 >>>[c00000001e1c7c80] [c0000000004b5054] .do_page_fault+0x440/0x624 >>>[c00000001e1c7e30] [c0000000000053fc] handle_page_fault+0x20/0x5c >>>Mem-info: >>>Node 0 DMA per-cpu: >>>CPU 0: hi: 6, btch: 1 usd: 1 >>>CPU 1: hi: 6, btch: 1 usd: 1 >>>Node 1 DMA per-cpu: >>>CPU 0: hi: 6, btch: 1 usd: 5 >>>CPU 1: hi: 6, btch: 1 usd: 5 >>>Active:31 inactive:2628 dirty:1 writeback:6 unstable:0 >>> free:156 slab:13071 mapped:548 pagetables:41109 bounce:0 >>>Node 0 DMA free:5312kB min:5760kB low:7168kB high:8640kB active:1024kB inactive:768kB present:3928832kB pages_scanned:3912 all_unreclaimable? no >>>lowmem_reserve[]: 0 0 0 >>>Node 1 DMA free:4672kB min:4992kB low:6208kB high:7488kB active:960kB inactive:169792kB present:3404992kB pages_scanned:140140 all_unreclaimable? no >>>lowmem_reserve[]: 0 0 0 >>>Node 0 DMA: 5*64kB 15*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB 0*8192kB 0*16384kB = 5312kB >>>Node 1 DMA: 8*64kB 5*128kB 6*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB 0*8192kB 0*16384kB = 4736kB >>>8185 total pagecache pages >>>Swap cache: add 29545, delete 21888, find 5602/10373 >>>Free swap = 803712kB >>>Total swap = 2048128kB >>>114688 pages of RAM >>>766 reserved pages >>>231218 pages shared >>>7657 pages swap cached >>>Out of memory: kill process 15371 (msgctl08) score 39223 or a child >>>Killed process 15373 (msgctl08) >>>INFO: task msgctl08:15385 blocked for more than 120 seconds. >>>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. >>>Call Trace: >>>[c0000000e9a43290] [0000000000000001] 0x1 (unreliable) >>>[c0000000e9a43460] [c000000000010acc] .__switch_to+0x128/0x16c >>>[c0000000e9a434f0] [c0000000004b07a4] .schedule+0x7ac/0x890 >>>[c0000000e9a435e0] [c0000000004b0d90] .io_schedule+0x7c/0xe8 >>>[c0000000e9a43670] [c0000000002d7f44] .get_request_wait+0x15c/0x1e0 >>>[c0000000e9a43750] [c0000000002d8950] .__make_request+0x3ec/0x4d8 >>>[c0000000e9a43800] [c0000000002d6624] .generic_make_request+0x35c/0x3b0 >>>[c0000000e9a438e0] [c0000000002d81b0] .submit_bio+0x118/0x140 >>>[c0000000e9a439a0] [c0000000000dfa88] .swap_readpage+0x94/0xb4 >>>[c0000000e9a43a20] [c0000000000e02b8] .read_swap_cache_async+0x110/0x160 >>>[c0000000e9a43ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 >>>[c0000000e9a43ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 >>>[c0000000e9a43c80] [c0000000004b5054] .do_page_fault+0x440/0x624 >>>[c0000000e9a43e30] [c0000000000053fc] handle_page_fault+0x20/0x5c >>>INFO: task msgctl08:15405 blocked for more than 120 seconds. >>>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. >>>Call Trace: >>>[c0000000b1757290] [0000000000000001] 0x1 (unreliable) >>>[c0000000b1757460] [c000000000010acc] .__switch_to+0x128/0x16c >>>[c0000000b17574f0] [c0000000004b07a4] .schedule+0x7ac/0x890 >>>[c0000000b17575e0] [c0000000004b0d90] .io_schedule+0x7c/0xe8 >>>[c0000000b1757670] [c0000000002d7f44] .get_request_wait+0x15c/0x1e0 >>>[c0000000b1757750] [c0000000002d8950] .__make_request+0x3ec/0x4d8 >>>[c0000000b1757800] [c0000000002d6624] .generic_make_request+0x35c/0x3b0 >>>[c0000000b17578e0] [c0000000002d81b0] .submit_bio+0x118/0x140 >>>[c0000000b17579a0] [c0000000000dfa88] .swap_readpage+0x94/0xb4 >>>[c0000000b1757a20] [c0000000000e02b8] .read_swap_cache_async+0x110/0x160 >>>[c0000000b1757ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 >>>[c0000000b1757ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 >>>[c0000000b1757c80] [c0000000004b5054] .do_page_fault+0x440/0x624 >>>[c0000000b1757e30] [c0000000000053fc] handle_page_fault+0x20/0x5c >>>INFO: task msgctl08:15406 blocked for more than 120 seconds. >>>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. >>>Call Trace: >>>[c0000000eec53290] [0000000000000001] 0x1 (unreliable) >>>[c0000000eec53460] [c000000000010acc] .__switch_to+0x128/0x16c >>>[c0000000eec534f0] [c0000000004b07a4] .schedule+0x7ac/0x890 >>>[c0000000eec535e0] [c0000000004b0d90] .io_schedule+0x7c/0xe8 >>>[c0000000eec53670] [c0000000002d7f44] .get_request_wait+0x15c/0x1e0 >>>[c0000000eec53750] [c0000000002d8950] .__make_request+0x3ec/0x4d8 >>>[c0000000eec53800] [c0000000002d6624] .generic_make_request+0x35c/0x3b0 >>>[c0000000eec538e0] [c0000000002d81b0] .submit_bio+0x118/0x140 >>>[c0000000eec539a0] [c0000000000dfa88] .swap_readpage+0x94/0xb4 >>>[c0000000eec53a20] [c0000000000e02b8] .read_swap_cache_async+0x110/0x160 >>>[c0000000eec53ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 >>>[c0000000eec53ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 >>>[c0000000eec53c80] [c0000000004b5054] .do_page_fault+0x440/0x624 >>>[c0000000eec53e30] [c0000000000053fc] handle_page_fault+0x20/0x5c Kamalesh, I'm wondering whether this is not related to a problem found in LTP after msgmni had been changed from 16 to a computed value. Does your ltp test suite contain the patch I had sent in thread http://lkml.org/lkml/2008/2/21/123? If you want to have a look at it, discussion started in thread http://lkml.org/lkml/2008/2/16/6 Regards, Nadia ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [BUG] 2.6.26-rc1-git4 - task blocked on powerpc for more than 120 seconds 2008-05-15 11:30 ` Nadia Derbey @ 2008-05-16 14:26 ` Kamalesh Babulal 2008-05-19 8:23 ` Nadia Derbey 0 siblings, 1 reply; 8+ messages in thread From: Kamalesh Babulal @ 2008-05-16 14:26 UTC (permalink / raw) To: Nadia Derbey; +Cc: linuxppc-dev, Adrian Bunk, Balbir Singh, kernel list Nadia Derbey wrote: > Kamalesh Babulal wrote: >> Adrian Bunk wrote: >> >>> On Wed, May 07, 2008 at 06:52:53PM +0530, Kamalesh Babulal wrote: >>> >>>> While running the ltp over the powerpc with the 2.6.26-rc1-git4 kernel, >>>> task get blocked for more 120 seconds and does not proceeds future. >>>> >>>> The task msgctl08 is a ipc testcase, which test the msgget(2) msgctl(2) >>>> syscalls. >>> Thanks for your report. >>> >>> I assume this is reproducible? >>> >>> If yes, what was the last kernel that worked for you? >> >> This is reproducible on 2.6.26-rc1 and 2.6.26-rc1-git(s). 2.6.25 is the last know kernel >> to work. >> >> >>> I've also added Pierre Peiffer and Nadia Derbey to the Cc since their >>> recent ipc commits are my first suspects. >>> >>> >>>> Machine is stuck in the task, printing the following call trace >>>> more than 5000 times. The oom-killer invoked once in-between. >>>> >>>> INFO: task msgctl08:16248 blocked for more than 120 seconds. >>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. >>>> Call Trace: >>>> [c0000000762435a0] [0000000000000001] 0x1 (unreliable) >>>> [c000000076243770] [c000000000010acc] .__switch_to+0x128/0x16c >>>> [c000000076243800] [c0000000004b07a4] .schedule+0x7ac/0x890 >>>> [c0000000762438f0] [c0000000004b2ba4] .rwsem_down_failed_common+0x260/0x2b0 >>>> [c0000000762439b0] [c0000000004b2c60] .rwsem_down_read_failed+0x2c/0x44 >>>> [c000000076243a60] [c0000000004b1b84] .down_read+0x44/0x5c >>>> [c000000076243af0] [c000000000295e10] .ipc_lock+0x34/0xe0 >>>> [c000000076243b90] [c00000000029690c] .ipc_lock_check+0x24/0x78 >>>> [c000000076243c20] [c0000000002972c0] .do_msgsnd+0xb0/0x3f8 >>>> [c000000076243d10] [c000000000293ce8] .compat_sys_msgsnd+0x94/0xc0 >>>> [c000000076243db0] [c000000000014418] .compat_sys_ipc+0x130/0x1f4 >>>> [c000000076243e30] [c000000000008734] syscall_exit+0x0/0x40 >>>> msgctl08 invoked oom-killer: gfp_mask=0x1200d2, order=0, oomkilladj=0 >>>> Call Trace: >>>> [c00000001e1c7640] [c0000000000101bc] .show_stack+0x70/0x1bc (unreliable) >>>> [c00000001e1c76f0] [c0000000000c2c78] .oom_kill_process+0x78/0x230 >>>> [c00000001e1c77a0] [c0000000000c3390] .out_of_memory+0x28c/0x320 >>>> [c00000001e1c7870] [c0000000000c70ac] .__alloc_pages_internal+0x364/0x468 >>>> [c00000001e1c7980] [c0000000000e83cc] .alloc_page_vma+0x120/0x14c >>>> [c00000001e1c7a20] [c0000000000e0224] .read_swap_cache_async+0x7c/0x160 >>>> [c00000001e1c7ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 >>>> [c00000001e1c7ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 >>>> [c00000001e1c7c80] [c0000000004b5054] .do_page_fault+0x440/0x624 >>>> [c00000001e1c7e30] [c0000000000053fc] handle_page_fault+0x20/0x5c >>>> Mem-info: >>>> Node 0 DMA per-cpu: >>>> CPU 0: hi: 6, btch: 1 usd: 1 >>>> CPU 1: hi: 6, btch: 1 usd: 1 >>>> Node 1 DMA per-cpu: >>>> CPU 0: hi: 6, btch: 1 usd: 5 >>>> CPU 1: hi: 6, btch: 1 usd: 5 >>>> Active:31 inactive:2628 dirty:1 writeback:6 unstable:0 >>>> free:156 slab:13071 mapped:548 pagetables:41109 bounce:0 >>>> Node 0 DMA free:5312kB min:5760kB low:7168kB high:8640kB active:1024kB inactive:768kB present:3928832kB pages_scanned:3912 all_unreclaimable? no >>>> lowmem_reserve[]: 0 0 0 >>>> Node 1 DMA free:4672kB min:4992kB low:6208kB high:7488kB active:960kB inactive:169792kB present:3404992kB pages_scanned:140140 all_unreclaimable? no >>>> lowmem_reserve[]: 0 0 0 >>>> Node 0 DMA: 5*64kB 15*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB 0*8192kB 0*16384kB = 5312kB >>>> Node 1 DMA: 8*64kB 5*128kB 6*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB 0*8192kB 0*16384kB = 4736kB >>>> 8185 total pagecache pages >>>> Swap cache: add 29545, delete 21888, find 5602/10373 >>>> Free swap = 803712kB >>>> Total swap = 2048128kB >>>> 114688 pages of RAM >>>> 766 reserved pages >>>> 231218 pages shared >>>> 7657 pages swap cached >>>> Out of memory: kill process 15371 (msgctl08) score 39223 or a child >>>> Killed process 15373 (msgctl08) >>>> INFO: task msgctl08:15385 blocked for more than 120 seconds. >>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. >>>> Call Trace: >>>> [c0000000e9a43290] [0000000000000001] 0x1 (unreliable) >>>> [c0000000e9a43460] [c000000000010acc] .__switch_to+0x128/0x16c >>>> [c0000000e9a434f0] [c0000000004b07a4] .schedule+0x7ac/0x890 >>>> [c0000000e9a435e0] [c0000000004b0d90] .io_schedule+0x7c/0xe8 >>>> [c0000000e9a43670] [c0000000002d7f44] .get_request_wait+0x15c/0x1e0 >>>> [c0000000e9a43750] [c0000000002d8950] .__make_request+0x3ec/0x4d8 >>>> [c0000000e9a43800] [c0000000002d6624] .generic_make_request+0x35c/0x3b0 >>>> [c0000000e9a438e0] [c0000000002d81b0] .submit_bio+0x118/0x140 >>>> [c0000000e9a439a0] [c0000000000dfa88] .swap_readpage+0x94/0xb4 >>>> [c0000000e9a43a20] [c0000000000e02b8] .read_swap_cache_async+0x110/0x160 >>>> [c0000000e9a43ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 >>>> [c0000000e9a43ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 >>>> [c0000000e9a43c80] [c0000000004b5054] .do_page_fault+0x440/0x624 >>>> [c0000000e9a43e30] [c0000000000053fc] handle_page_fault+0x20/0x5c >>>> INFO: task msgctl08:15405 blocked for more than 120 seconds. >>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. >>>> Call Trace: >>>> [c0000000b1757290] [0000000000000001] 0x1 (unreliable) >>>> [c0000000b1757460] [c000000000010acc] .__switch_to+0x128/0x16c >>>> [c0000000b17574f0] [c0000000004b07a4] .schedule+0x7ac/0x890 >>>> [c0000000b17575e0] [c0000000004b0d90] .io_schedule+0x7c/0xe8 >>>> [c0000000b1757670] [c0000000002d7f44] .get_request_wait+0x15c/0x1e0 >>>> [c0000000b1757750] [c0000000002d8950] .__make_request+0x3ec/0x4d8 >>>> [c0000000b1757800] [c0000000002d6624] .generic_make_request+0x35c/0x3b0 >>>> [c0000000b17578e0] [c0000000002d81b0] .submit_bio+0x118/0x140 >>>> [c0000000b17579a0] [c0000000000dfa88] .swap_readpage+0x94/0xb4 >>>> [c0000000b1757a20] [c0000000000e02b8] .read_swap_cache_async+0x110/0x160 >>>> [c0000000b1757ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 >>>> [c0000000b1757ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 >>>> [c0000000b1757c80] [c0000000004b5054] .do_page_fault+0x440/0x624 >>>> [c0000000b1757e30] [c0000000000053fc] handle_page_fault+0x20/0x5c >>>> INFO: task msgctl08:15406 blocked for more than 120 seconds. >>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. >>>> Call Trace: >>>> [c0000000eec53290] [0000000000000001] 0x1 (unreliable) >>>> [c0000000eec53460] [c000000000010acc] .__switch_to+0x128/0x16c >>>> [c0000000eec534f0] [c0000000004b07a4] .schedule+0x7ac/0x890 >>>> [c0000000eec535e0] [c0000000004b0d90] .io_schedule+0x7c/0xe8 >>>> [c0000000eec53670] [c0000000002d7f44] .get_request_wait+0x15c/0x1e0 >>>> [c0000000eec53750] [c0000000002d8950] .__make_request+0x3ec/0x4d8 >>>> [c0000000eec53800] [c0000000002d6624] .generic_make_request+0x35c/0x3b0 >>>> [c0000000eec538e0] [c0000000002d81b0] .submit_bio+0x118/0x140 >>>> [c0000000eec539a0] [c0000000000dfa88] .swap_readpage+0x94/0xb4 >>>> [c0000000eec53a20] [c0000000000e02b8] .read_swap_cache_async+0x110/0x160 >>>> [c0000000eec53ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 >>>> [c0000000eec53ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 >>>> [c0000000eec53c80] [c0000000004b5054] .do_page_fault+0x440/0x624 >>>> [c0000000eec53e30] [c0000000000053fc] handle_page_fault+0x20/0x5c > > Kamalesh, > > I'm wondering whether this is not related to a problem found in LTP > after msgmni had been changed from 16 to a computed value. > > Does your ltp test suite contain the patch I had sent in thread > http://lkml.org/lkml/2008/2/21/123? > > If you want to have a look at it, discussion started in thread > http://lkml.org/lkml/2008/2/16/6 > > Regards, > Nadia > Hi Nadia, Thanks, I am using the LTP-20080430 and tried it with the 2.6.26-rc2-git4 kernel. but the call trace remains the same. -- Thanks & Regards, Kamalesh Babulal, Linux Technology Center, IBM, ISTL. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [BUG] 2.6.26-rc1-git4 - task blocked on powerpc for more than 120 seconds 2008-05-16 14:26 ` Kamalesh Babulal @ 2008-05-19 8:23 ` Nadia Derbey 2008-05-19 10:14 ` Kamalesh Babulal 2008-05-19 10:31 ` Nadia Derbey 0 siblings, 2 replies; 8+ messages in thread From: Nadia Derbey @ 2008-05-19 8:23 UTC (permalink / raw) To: Kamalesh Babulal; +Cc: linuxppc-dev, Adrian Bunk, Balbir Singh, kernel list Kamalesh Babulal wrote: > Nadia Derbey wrote: > >>Kamalesh Babulal wrote: >> >>>Adrian Bunk wrote: >>> >>> >>>>On Wed, May 07, 2008 at 06:52:53PM +0530, Kamalesh Babulal wrote: >>>> >>>> >>>>>While running the ltp over the powerpc with the 2.6.26-rc1-git4 kernel, >>>>>task get blocked for more 120 seconds and does not proceeds future. >>>>> >>>>>The task msgctl08 is a ipc testcase, which test the msgget(2) msgctl(2) >>>>>syscalls. >>>> >>>>Thanks for your report. >>>> >>>>I assume this is reproducible? >>>> >>>>If yes, what was the last kernel that worked for you? >>> >>>This is reproducible on 2.6.26-rc1 and 2.6.26-rc1-git(s). 2.6.25 is the last know kernel >>>to work. >>> >>> >>> >>>>I've also added Pierre Peiffer and Nadia Derbey to the Cc since their >>>>recent ipc commits are my first suspects. >>>> >>>> >>>> >>>>>Machine is stuck in the task, printing the following call trace >>>>>more than 5000 times. The oom-killer invoked once in-between. >>>>> >>>>>INFO: task msgctl08:16248 blocked for more than 120 seconds. >>>>>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. >>>>>Call Trace: >>>>>[c0000000762435a0] [0000000000000001] 0x1 (unreliable) >>>>>[c000000076243770] [c000000000010acc] .__switch_to+0x128/0x16c >>>>>[c000000076243800] [c0000000004b07a4] .schedule+0x7ac/0x890 >>>>>[c0000000762438f0] [c0000000004b2ba4] .rwsem_down_failed_common+0x260/0x2b0 >>>>>[c0000000762439b0] [c0000000004b2c60] .rwsem_down_read_failed+0x2c/0x44 >>>>>[c000000076243a60] [c0000000004b1b84] .down_read+0x44/0x5c >>>>>[c000000076243af0] [c000000000295e10] .ipc_lock+0x34/0xe0 >>>>>[c000000076243b90] [c00000000029690c] .ipc_lock_check+0x24/0x78 >>>>>[c000000076243c20] [c0000000002972c0] .do_msgsnd+0xb0/0x3f8 >>>>>[c000000076243d10] [c000000000293ce8] .compat_sys_msgsnd+0x94/0xc0 >>>>>[c000000076243db0] [c000000000014418] .compat_sys_ipc+0x130/0x1f4 >>>>>[c000000076243e30] [c000000000008734] syscall_exit+0x0/0x40 >>>>>msgctl08 invoked oom-killer: gfp_mask=0x1200d2, order=0, oomkilladj=0 >>>>>Call Trace: >>>>>[c00000001e1c7640] [c0000000000101bc] .show_stack+0x70/0x1bc (unreliable) >>>>>[c00000001e1c76f0] [c0000000000c2c78] .oom_kill_process+0x78/0x230 >>>>>[c00000001e1c77a0] [c0000000000c3390] .out_of_memory+0x28c/0x320 >>>>>[c00000001e1c7870] [c0000000000c70ac] .__alloc_pages_internal+0x364/0x468 >>>>>[c00000001e1c7980] [c0000000000e83cc] .alloc_page_vma+0x120/0x14c >>>>>[c00000001e1c7a20] [c0000000000e0224] .read_swap_cache_async+0x7c/0x160 >>>>>[c00000001e1c7ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 >>>>>[c00000001e1c7ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 >>>>>[c00000001e1c7c80] [c0000000004b5054] .do_page_fault+0x440/0x624 >>>>>[c00000001e1c7e30] [c0000000000053fc] handle_page_fault+0x20/0x5c >>>>>Mem-info: >>>>>Node 0 DMA per-cpu: >>>>>CPU 0: hi: 6, btch: 1 usd: 1 >>>>>CPU 1: hi: 6, btch: 1 usd: 1 >>>>>Node 1 DMA per-cpu: >>>>>CPU 0: hi: 6, btch: 1 usd: 5 >>>>>CPU 1: hi: 6, btch: 1 usd: 5 >>>>>Active:31 inactive:2628 dirty:1 writeback:6 unstable:0 >>>>>free:156 slab:13071 mapped:548 pagetables:41109 bounce:0 >>>>>Node 0 DMA free:5312kB min:5760kB low:7168kB high:8640kB active:1024kB inactive:768kB present:3928832kB pages_scanned:3912 all_unreclaimable? no >>>>>lowmem_reserve[]: 0 0 0 >>>>>Node 1 DMA free:4672kB min:4992kB low:6208kB high:7488kB active:960kB inactive:169792kB present:3404992kB pages_scanned:140140 all_unreclaimable? no >>>>>lowmem_reserve[]: 0 0 0 >>>>>Node 0 DMA: 5*64kB 15*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB 0*8192kB 0*16384kB = 5312kB >>>>>Node 1 DMA: 8*64kB 5*128kB 6*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB 0*8192kB 0*16384kB = 4736kB >>>>>8185 total pagecache pages >>>>>Swap cache: add 29545, delete 21888, find 5602/10373 >>>>>Free swap = 803712kB >>>>>Total swap = 2048128kB >>>>>114688 pages of RAM >>>>>766 reserved pages >>>>>231218 pages shared >>>>>7657 pages swap cached >>>>>Out of memory: kill process 15371 (msgctl08) score 39223 or a child >>>>>Killed process 15373 (msgctl08) >>>>>INFO: task msgctl08:15385 blocked for more than 120 seconds. >>>>>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. >>>>>Call Trace: >>>>>[c0000000e9a43290] [0000000000000001] 0x1 (unreliable) >>>>>[c0000000e9a43460] [c000000000010acc] .__switch_to+0x128/0x16c >>>>>[c0000000e9a434f0] [c0000000004b07a4] .schedule+0x7ac/0x890 >>>>>[c0000000e9a435e0] [c0000000004b0d90] .io_schedule+0x7c/0xe8 >>>>>[c0000000e9a43670] [c0000000002d7f44] .get_request_wait+0x15c/0x1e0 >>>>>[c0000000e9a43750] [c0000000002d8950] .__make_request+0x3ec/0x4d8 >>>>>[c0000000e9a43800] [c0000000002d6624] .generic_make_request+0x35c/0x3b0 >>>>>[c0000000e9a438e0] [c0000000002d81b0] .submit_bio+0x118/0x140 >>>>>[c0000000e9a439a0] [c0000000000dfa88] .swap_readpage+0x94/0xb4 >>>>>[c0000000e9a43a20] [c0000000000e02b8] .read_swap_cache_async+0x110/0x160 >>>>>[c0000000e9a43ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 >>>>>[c0000000e9a43ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 >>>>>[c0000000e9a43c80] [c0000000004b5054] .do_page_fault+0x440/0x624 >>>>>[c0000000e9a43e30] [c0000000000053fc] handle_page_fault+0x20/0x5c >>>>>INFO: task msgctl08:15405 blocked for more than 120 seconds. >>>>>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. >>>>>Call Trace: >>>>>[c0000000b1757290] [0000000000000001] 0x1 (unreliable) >>>>>[c0000000b1757460] [c000000000010acc] .__switch_to+0x128/0x16c >>>>>[c0000000b17574f0] [c0000000004b07a4] .schedule+0x7ac/0x890 >>>>>[c0000000b17575e0] [c0000000004b0d90] .io_schedule+0x7c/0xe8 >>>>>[c0000000b1757670] [c0000000002d7f44] .get_request_wait+0x15c/0x1e0 >>>>>[c0000000b1757750] [c0000000002d8950] .__make_request+0x3ec/0x4d8 >>>>>[c0000000b1757800] [c0000000002d6624] .generic_make_request+0x35c/0x3b0 >>>>>[c0000000b17578e0] [c0000000002d81b0] .submit_bio+0x118/0x140 >>>>>[c0000000b17579a0] [c0000000000dfa88] .swap_readpage+0x94/0xb4 >>>>>[c0000000b1757a20] [c0000000000e02b8] .read_swap_cache_async+0x110/0x160 >>>>>[c0000000b1757ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 >>>>>[c0000000b1757ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 >>>>>[c0000000b1757c80] [c0000000004b5054] .do_page_fault+0x440/0x624 >>>>>[c0000000b1757e30] [c0000000000053fc] handle_page_fault+0x20/0x5c >>>>>INFO: task msgctl08:15406 blocked for more than 120 seconds. >>>>>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. >>>>>Call Trace: >>>>>[c0000000eec53290] [0000000000000001] 0x1 (unreliable) >>>>>[c0000000eec53460] [c000000000010acc] .__switch_to+0x128/0x16c >>>>>[c0000000eec534f0] [c0000000004b07a4] .schedule+0x7ac/0x890 >>>>>[c0000000eec535e0] [c0000000004b0d90] .io_schedule+0x7c/0xe8 >>>>>[c0000000eec53670] [c0000000002d7f44] .get_request_wait+0x15c/0x1e0 >>>>>[c0000000eec53750] [c0000000002d8950] .__make_request+0x3ec/0x4d8 >>>>>[c0000000eec53800] [c0000000002d6624] .generic_make_request+0x35c/0x3b0 >>>>>[c0000000eec538e0] [c0000000002d81b0] .submit_bio+0x118/0x140 >>>>>[c0000000eec539a0] [c0000000000dfa88] .swap_readpage+0x94/0xb4 >>>>>[c0000000eec53a20] [c0000000000e02b8] .read_swap_cache_async+0x110/0x160 >>>>>[c0000000eec53ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 >>>>>[c0000000eec53ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 >>>>>[c0000000eec53c80] [c0000000004b5054] .do_page_fault+0x440/0x624 >>>>>[c0000000eec53e30] [c0000000000053fc] handle_page_fault+0x20/0x5c >> >>Kamalesh, >> >>I'm wondering whether this is not related to a problem found in LTP >>after msgmni had been changed from 16 to a computed value. >> >>Does your ltp test suite contain the patch I had sent in thread >>http://lkml.org/lkml/2008/2/21/123? >> >>If you want to have a look at it, discussion started in thread >>http://lkml.org/lkml/2008/2/16/6 >> >>Regards, >>Nadia >> > > Hi Nadia, > Thanks, I am using the LTP-20080430 and tried it with the 2.6.26-rc2-git4 kernel. > but the call trace remains the same. > Ok, I'll try to find a ppc machine: could not reproduce it on my Xeon machine (but actually it was with a 2.6.26-rc1 kernel). Regards, Nadia ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [BUG] 2.6.26-rc1-git4 - task blocked on powerpc for more than 120 seconds 2008-05-19 8:23 ` Nadia Derbey @ 2008-05-19 10:14 ` Kamalesh Babulal 2008-05-19 10:31 ` Nadia Derbey 1 sibling, 0 replies; 8+ messages in thread From: Kamalesh Babulal @ 2008-05-19 10:14 UTC (permalink / raw) To: Nadia Derbey; +Cc: linuxppc-dev, Adrian Bunk, Balbir Singh, kernel list Nadia Derbey wrote: > Kamalesh Babulal wrote: >> Nadia Derbey wrote: >> >>> Kamalesh Babulal wrote: >>> >>>> Adrian Bunk wrote: >>>> >>>> >>>>> On Wed, May 07, 2008 at 06:52:53PM +0530, Kamalesh Babulal wrote: >>>>> >>>>> >>>>>> While running the ltp over the powerpc with the 2.6.26-rc1-git4 kernel, >>>>>> task get blocked for more 120 seconds and does not proceeds future. >>>>>> >>>>>> The task msgctl08 is a ipc testcase, which test the msgget(2) msgctl(2) >>>>>> syscalls. >>>>> Thanks for your report. >>>>> >>>>> I assume this is reproducible? >>>>> >>>>> If yes, what was the last kernel that worked for you? >>>> This is reproducible on 2.6.26-rc1 and 2.6.26-rc1-git(s). 2.6.25 is the last know kernel >>>> to work. >>>> >>>> >>>> >>>>> I've also added Pierre Peiffer and Nadia Derbey to the Cc since their >>>>> recent ipc commits are my first suspects. >>>>> >>>>> >>>>> >>>>>> Machine is stuck in the task, printing the following call trace >>>>>> more than 5000 times. The oom-killer invoked once in-between. >>>>>> >>>>>> INFO: task msgctl08:16248 blocked for more than 120 seconds. >>>>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. >>>>>> Call Trace: >>>>>> [c0000000762435a0] [0000000000000001] 0x1 (unreliable) >>>>>> [c000000076243770] [c000000000010acc] .__switch_to+0x128/0x16c >>>>>> [c000000076243800] [c0000000004b07a4] .schedule+0x7ac/0x890 >>>>>> [c0000000762438f0] [c0000000004b2ba4] .rwsem_down_failed_common+0x260/0x2b0 >>>>>> [c0000000762439b0] [c0000000004b2c60] .rwsem_down_read_failed+0x2c/0x44 >>>>>> [c000000076243a60] [c0000000004b1b84] .down_read+0x44/0x5c >>>>>> [c000000076243af0] [c000000000295e10] .ipc_lock+0x34/0xe0 >>>>>> [c000000076243b90] [c00000000029690c] .ipc_lock_check+0x24/0x78 >>>>>> [c000000076243c20] [c0000000002972c0] .do_msgsnd+0xb0/0x3f8 >>>>>> [c000000076243d10] [c000000000293ce8] .compat_sys_msgsnd+0x94/0xc0 >>>>>> [c000000076243db0] [c000000000014418] .compat_sys_ipc+0x130/0x1f4 >>>>>> [c000000076243e30] [c000000000008734] syscall_exit+0x0/0x40 >>>>>> msgctl08 invoked oom-killer: gfp_mask=0x1200d2, order=0, oomkilladj=0 >>>>>> Call Trace: >>>>>> [c00000001e1c7640] [c0000000000101bc] .show_stack+0x70/0x1bc (unreliable) >>>>>> [c00000001e1c76f0] [c0000000000c2c78] .oom_kill_process+0x78/0x230 >>>>>> [c00000001e1c77a0] [c0000000000c3390] .out_of_memory+0x28c/0x320 >>>>>> [c00000001e1c7870] [c0000000000c70ac] .__alloc_pages_internal+0x364/0x468 >>>>>> [c00000001e1c7980] [c0000000000e83cc] .alloc_page_vma+0x120/0x14c >>>>>> [c00000001e1c7a20] [c0000000000e0224] .read_swap_cache_async+0x7c/0x160 >>>>>> [c00000001e1c7ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 >>>>>> [c00000001e1c7ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 >>>>>> [c00000001e1c7c80] [c0000000004b5054] .do_page_fault+0x440/0x624 >>>>>> [c00000001e1c7e30] [c0000000000053fc] handle_page_fault+0x20/0x5c >>>>>> Mem-info: >>>>>> Node 0 DMA per-cpu: >>>>>> CPU 0: hi: 6, btch: 1 usd: 1 >>>>>> CPU 1: hi: 6, btch: 1 usd: 1 >>>>>> Node 1 DMA per-cpu: >>>>>> CPU 0: hi: 6, btch: 1 usd: 5 >>>>>> CPU 1: hi: 6, btch: 1 usd: 5 >>>>>> Active:31 inactive:2628 dirty:1 writeback:6 unstable:0 >>>>>> free:156 slab:13071 mapped:548 pagetables:41109 bounce:0 >>>>>> Node 0 DMA free:5312kB min:5760kB low:7168kB high:8640kB active:1024kB inactive:768kB present:3928832kB pages_scanned:3912 all_unreclaimable? no >>>>>> lowmem_reserve[]: 0 0 0 >>>>>> Node 1 DMA free:4672kB min:4992kB low:6208kB high:7488kB active:960kB inactive:169792kB present:3404992kB pages_scanned:140140 all_unreclaimable? no >>>>>> lowmem_reserve[]: 0 0 0 >>>>>> Node 0 DMA: 5*64kB 15*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB 0*8192kB 0*16384kB = 5312kB >>>>>> Node 1 DMA: 8*64kB 5*128kB 6*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB 0*8192kB 0*16384kB = 4736kB >>>>>> 8185 total pagecache pages >>>>>> Swap cache: add 29545, delete 21888, find 5602/10373 >>>>>> Free swap = 803712kB >>>>>> Total swap = 2048128kB >>>>>> 114688 pages of RAM >>>>>> 766 reserved pages >>>>>> 231218 pages shared >>>>>> 7657 pages swap cached >>>>>> Out of memory: kill process 15371 (msgctl08) score 39223 or a child >>>>>> Killed process 15373 (msgctl08) >>>>>> INFO: task msgctl08:15385 blocked for more than 120 seconds. >>>>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. >>>>>> Call Trace: >>>>>> [c0000000e9a43290] [0000000000000001] 0x1 (unreliable) >>>>>> [c0000000e9a43460] [c000000000010acc] .__switch_to+0x128/0x16c >>>>>> [c0000000e9a434f0] [c0000000004b07a4] .schedule+0x7ac/0x890 >>>>>> [c0000000e9a435e0] [c0000000004b0d90] .io_schedule+0x7c/0xe8 >>>>>> [c0000000e9a43670] [c0000000002d7f44] .get_request_wait+0x15c/0x1e0 >>>>>> [c0000000e9a43750] [c0000000002d8950] .__make_request+0x3ec/0x4d8 >>>>>> [c0000000e9a43800] [c0000000002d6624] .generic_make_request+0x35c/0x3b0 >>>>>> [c0000000e9a438e0] [c0000000002d81b0] .submit_bio+0x118/0x140 >>>>>> [c0000000e9a439a0] [c0000000000dfa88] .swap_readpage+0x94/0xb4 >>>>>> [c0000000e9a43a20] [c0000000000e02b8] .read_swap_cache_async+0x110/0x160 >>>>>> [c0000000e9a43ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 >>>>>> [c0000000e9a43ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 >>>>>> [c0000000e9a43c80] [c0000000004b5054] .do_page_fault+0x440/0x624 >>>>>> [c0000000e9a43e30] [c0000000000053fc] handle_page_fault+0x20/0x5c >>>>>> INFO: task msgctl08:15405 blocked for more than 120 seconds. >>>>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. >>>>>> Call Trace: >>>>>> [c0000000b1757290] [0000000000000001] 0x1 (unreliable) >>>>>> [c0000000b1757460] [c000000000010acc] .__switch_to+0x128/0x16c >>>>>> [c0000000b17574f0] [c0000000004b07a4] .schedule+0x7ac/0x890 >>>>>> [c0000000b17575e0] [c0000000004b0d90] .io_schedule+0x7c/0xe8 >>>>>> [c0000000b1757670] [c0000000002d7f44] .get_request_wait+0x15c/0x1e0 >>>>>> [c0000000b1757750] [c0000000002d8950] .__make_request+0x3ec/0x4d8 >>>>>> [c0000000b1757800] [c0000000002d6624] .generic_make_request+0x35c/0x3b0 >>>>>> [c0000000b17578e0] [c0000000002d81b0] .submit_bio+0x118/0x140 >>>>>> [c0000000b17579a0] [c0000000000dfa88] .swap_readpage+0x94/0xb4 >>>>>> [c0000000b1757a20] [c0000000000e02b8] .read_swap_cache_async+0x110/0x160 >>>>>> [c0000000b1757ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 >>>>>> [c0000000b1757ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 >>>>>> [c0000000b1757c80] [c0000000004b5054] .do_page_fault+0x440/0x624 >>>>>> [c0000000b1757e30] [c0000000000053fc] handle_page_fault+0x20/0x5c >>>>>> INFO: task msgctl08:15406 blocked for more than 120 seconds. >>>>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. >>>>>> Call Trace: >>>>>> [c0000000eec53290] [0000000000000001] 0x1 (unreliable) >>>>>> [c0000000eec53460] [c000000000010acc] .__switch_to+0x128/0x16c >>>>>> [c0000000eec534f0] [c0000000004b07a4] .schedule+0x7ac/0x890 >>>>>> [c0000000eec535e0] [c0000000004b0d90] .io_schedule+0x7c/0xe8 >>>>>> [c0000000eec53670] [c0000000002d7f44] .get_request_wait+0x15c/0x1e0 >>>>>> [c0000000eec53750] [c0000000002d8950] .__make_request+0x3ec/0x4d8 >>>>>> [c0000000eec53800] [c0000000002d6624] .generic_make_request+0x35c/0x3b0 >>>>>> [c0000000eec538e0] [c0000000002d81b0] .submit_bio+0x118/0x140 >>>>>> [c0000000eec539a0] [c0000000000dfa88] .swap_readpage+0x94/0xb4 >>>>>> [c0000000eec53a20] [c0000000000e02b8] .read_swap_cache_async+0x110/0x160 >>>>>> [c0000000eec53ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 >>>>>> [c0000000eec53ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 >>>>>> [c0000000eec53c80] [c0000000004b5054] .do_page_fault+0x440/0x624 >>>>>> [c0000000eec53e30] [c0000000000053fc] handle_page_fault+0x20/0x5c >>> Kamalesh, >>> >>> I'm wondering whether this is not related to a problem found in LTP >>> after msgmni had been changed from 16 to a computed value. >>> >>> Does your ltp test suite contain the patch I had sent in thread >>> http://lkml.org/lkml/2008/2/21/123? >>> >>> If you want to have a look at it, discussion started in thread >>> http://lkml.org/lkml/2008/2/16/6 >>> >>> Regards, >>> Nadia >>> >> Hi Nadia, >> Thanks, I am using the LTP-20080430 and tried it with the 2.6.26-rc2-git4 kernel. >> but the call trace remains the same. >> > > Ok, I'll try to find a ppc machine: could not reproduce it on my Xeon > machine (but actually it was with a 2.6.26-rc1 kernel). > > Regards, > Nadia > Hi Nadia, The kernel tested with the LTP-20080430 version of LTP, does not produce the call trace. Sorry my fault on problem with the test setup at my end. I have updated the same in the morning http://lkml.org/lkml/2008/5/19/32. -- Thanks & Regards, Kamalesh Babulal, Linux Technology Center, IBM, ISTL. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [BUG] 2.6.26-rc1-git4 - task blocked on powerpc for more than 120 seconds 2008-05-19 8:23 ` Nadia Derbey 2008-05-19 10:14 ` Kamalesh Babulal @ 2008-05-19 10:31 ` Nadia Derbey 1 sibling, 0 replies; 8+ messages in thread From: Nadia Derbey @ 2008-05-19 10:31 UTC (permalink / raw) To: Nadia Derbey Cc: Adrian Bunk, kernel list, Kamalesh Babulal, linuxppc-dev, Balbir Singh [-- Attachment #1: Type: text/plain, Size: 8815 bytes --] Nadia Derbey wrote: > Kamalesh Babulal wrote: > >> Nadia Derbey wrote: >> >>> Kamalesh Babulal wrote: >>> >>>> Adrian Bunk wrote: >>>> >>>> >>>>> On Wed, May 07, 2008 at 06:52:53PM +0530, Kamalesh Babulal wrote: >>>>> >>>>> >>>>>> While running the ltp over the powerpc with the 2.6.26-rc1-git4 >>>>>> kernel, >>>>>> task get blocked for more 120 seconds and does not proceeds future. >>>>>> >>>>>> The task msgctl08 is a ipc testcase, which test the msgget(2) >>>>>> msgctl(2) >>>>>> syscalls. >>>>> >>>>> >>>>> Thanks for your report. >>>>> >>>>> I assume this is reproducible? >>>>> >>>>> If yes, what was the last kernel that worked for you? >>>> >>>> >>>> This is reproducible on 2.6.26-rc1 and 2.6.26-rc1-git(s). 2.6.25 is >>>> the last know kernel >>>> to work. >>>> >>>> >>>> >>>>> I've also added Pierre Peiffer and Nadia Derbey to the Cc since >>>>> their recent ipc commits are my first suspects. >>>>> >>>>> >>>>> >>>>>> Machine is stuck in the task, printing the following call trace >>>>>> more than 5000 times. The oom-killer invoked once in-between. >>>>>> >>>>>> INFO: task msgctl08:16248 blocked for more than 120 seconds. >>>>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >>>>>> message. >>>>>> Call Trace: >>>>>> [c0000000762435a0] [0000000000000001] 0x1 (unreliable) >>>>>> [c000000076243770] [c000000000010acc] .__switch_to+0x128/0x16c >>>>>> [c000000076243800] [c0000000004b07a4] .schedule+0x7ac/0x890 >>>>>> [c0000000762438f0] [c0000000004b2ba4] >>>>>> .rwsem_down_failed_common+0x260/0x2b0 >>>>>> [c0000000762439b0] [c0000000004b2c60] >>>>>> .rwsem_down_read_failed+0x2c/0x44 >>>>>> [c000000076243a60] [c0000000004b1b84] .down_read+0x44/0x5c >>>>>> [c000000076243af0] [c000000000295e10] .ipc_lock+0x34/0xe0 >>>>>> [c000000076243b90] [c00000000029690c] .ipc_lock_check+0x24/0x78 >>>>>> [c000000076243c20] [c0000000002972c0] .do_msgsnd+0xb0/0x3f8 >>>>>> [c000000076243d10] [c000000000293ce8] .compat_sys_msgsnd+0x94/0xc0 >>>>>> [c000000076243db0] [c000000000014418] .compat_sys_ipc+0x130/0x1f4 >>>>>> [c000000076243e30] [c000000000008734] syscall_exit+0x0/0x40 >>>>>> msgctl08 invoked oom-killer: gfp_mask=0x1200d2, order=0, oomkilladj=0 >>>>>> Call Trace: >>>>>> [c00000001e1c7640] [c0000000000101bc] .show_stack+0x70/0x1bc >>>>>> (unreliable) >>>>>> [c00000001e1c76f0] [c0000000000c2c78] .oom_kill_process+0x78/0x230 >>>>>> [c00000001e1c77a0] [c0000000000c3390] .out_of_memory+0x28c/0x320 >>>>>> [c00000001e1c7870] [c0000000000c70ac] >>>>>> .__alloc_pages_internal+0x364/0x468 >>>>>> [c00000001e1c7980] [c0000000000e83cc] .alloc_page_vma+0x120/0x14c >>>>>> [c00000001e1c7a20] [c0000000000e0224] >>>>>> .read_swap_cache_async+0x7c/0x160 >>>>>> [c00000001e1c7ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 >>>>>> [c00000001e1c7ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 >>>>>> [c00000001e1c7c80] [c0000000004b5054] .do_page_fault+0x440/0x624 >>>>>> [c00000001e1c7e30] [c0000000000053fc] handle_page_fault+0x20/0x5c >>>>>> Mem-info: >>>>>> Node 0 DMA per-cpu: >>>>>> CPU 0: hi: 6, btch: 1 usd: 1 >>>>>> CPU 1: hi: 6, btch: 1 usd: 1 >>>>>> Node 1 DMA per-cpu: >>>>>> CPU 0: hi: 6, btch: 1 usd: 5 >>>>>> CPU 1: hi: 6, btch: 1 usd: 5 >>>>>> Active:31 inactive:2628 dirty:1 writeback:6 unstable:0 >>>>>> free:156 slab:13071 mapped:548 pagetables:41109 bounce:0 >>>>>> Node 0 DMA free:5312kB min:5760kB low:7168kB high:8640kB >>>>>> active:1024kB inactive:768kB present:3928832kB pages_scanned:3912 >>>>>> all_unreclaimable? no >>>>>> lowmem_reserve[]: 0 0 0 >>>>>> Node 1 DMA free:4672kB min:4992kB low:6208kB high:7488kB >>>>>> active:960kB inactive:169792kB present:3404992kB >>>>>> pages_scanned:140140 all_unreclaimable? no >>>>>> lowmem_reserve[]: 0 0 0 >>>>>> Node 0 DMA: 5*64kB 15*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB >>>>>> 0*4096kB 0*8192kB 0*16384kB = 5312kB >>>>>> Node 1 DMA: 8*64kB 5*128kB 6*256kB 0*512kB 0*1024kB 1*2048kB >>>>>> 0*4096kB 0*8192kB 0*16384kB = 4736kB >>>>>> 8185 total pagecache pages >>>>>> Swap cache: add 29545, delete 21888, find 5602/10373 >>>>>> Free swap = 803712kB >>>>>> Total swap = 2048128kB >>>>>> 114688 pages of RAM >>>>>> 766 reserved pages >>>>>> 231218 pages shared >>>>>> 7657 pages swap cached >>>>>> Out of memory: kill process 15371 (msgctl08) score 39223 or a child >>>>>> Killed process 15373 (msgctl08) >>>>>> INFO: task msgctl08:15385 blocked for more than 120 seconds. >>>>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >>>>>> message. >>>>>> Call Trace: >>>>>> [c0000000e9a43290] [0000000000000001] 0x1 (unreliable) >>>>>> [c0000000e9a43460] [c000000000010acc] .__switch_to+0x128/0x16c >>>>>> [c0000000e9a434f0] [c0000000004b07a4] .schedule+0x7ac/0x890 >>>>>> [c0000000e9a435e0] [c0000000004b0d90] .io_schedule+0x7c/0xe8 >>>>>> [c0000000e9a43670] [c0000000002d7f44] .get_request_wait+0x15c/0x1e0 >>>>>> [c0000000e9a43750] [c0000000002d8950] .__make_request+0x3ec/0x4d8 >>>>>> [c0000000e9a43800] [c0000000002d6624] >>>>>> .generic_make_request+0x35c/0x3b0 >>>>>> [c0000000e9a438e0] [c0000000002d81b0] .submit_bio+0x118/0x140 >>>>>> [c0000000e9a439a0] [c0000000000dfa88] .swap_readpage+0x94/0xb4 >>>>>> [c0000000e9a43a20] [c0000000000e02b8] >>>>>> .read_swap_cache_async+0x110/0x160 >>>>>> [c0000000e9a43ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 >>>>>> [c0000000e9a43ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 >>>>>> [c0000000e9a43c80] [c0000000004b5054] .do_page_fault+0x440/0x624 >>>>>> [c0000000e9a43e30] [c0000000000053fc] handle_page_fault+0x20/0x5c >>>>>> INFO: task msgctl08:15405 blocked for more than 120 seconds. >>>>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >>>>>> message. >>>>>> Call Trace: >>>>>> [c0000000b1757290] [0000000000000001] 0x1 (unreliable) >>>>>> [c0000000b1757460] [c000000000010acc] .__switch_to+0x128/0x16c >>>>>> [c0000000b17574f0] [c0000000004b07a4] .schedule+0x7ac/0x890 >>>>>> [c0000000b17575e0] [c0000000004b0d90] .io_schedule+0x7c/0xe8 >>>>>> [c0000000b1757670] [c0000000002d7f44] .get_request_wait+0x15c/0x1e0 >>>>>> [c0000000b1757750] [c0000000002d8950] .__make_request+0x3ec/0x4d8 >>>>>> [c0000000b1757800] [c0000000002d6624] >>>>>> .generic_make_request+0x35c/0x3b0 >>>>>> [c0000000b17578e0] [c0000000002d81b0] .submit_bio+0x118/0x140 >>>>>> [c0000000b17579a0] [c0000000000dfa88] .swap_readpage+0x94/0xb4 >>>>>> [c0000000b1757a20] [c0000000000e02b8] >>>>>> .read_swap_cache_async+0x110/0x160 >>>>>> [c0000000b1757ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 >>>>>> [c0000000b1757ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 >>>>>> [c0000000b1757c80] [c0000000004b5054] .do_page_fault+0x440/0x624 >>>>>> [c0000000b1757e30] [c0000000000053fc] handle_page_fault+0x20/0x5c >>>>>> INFO: task msgctl08:15406 blocked for more than 120 seconds. >>>>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this >>>>>> message. >>>>>> Call Trace: >>>>>> [c0000000eec53290] [0000000000000001] 0x1 (unreliable) >>>>>> [c0000000eec53460] [c000000000010acc] .__switch_to+0x128/0x16c >>>>>> [c0000000eec534f0] [c0000000004b07a4] .schedule+0x7ac/0x890 >>>>>> [c0000000eec535e0] [c0000000004b0d90] .io_schedule+0x7c/0xe8 >>>>>> [c0000000eec53670] [c0000000002d7f44] .get_request_wait+0x15c/0x1e0 >>>>>> [c0000000eec53750] [c0000000002d8950] .__make_request+0x3ec/0x4d8 >>>>>> [c0000000eec53800] [c0000000002d6624] >>>>>> .generic_make_request+0x35c/0x3b0 >>>>>> [c0000000eec538e0] [c0000000002d81b0] .submit_bio+0x118/0x140 >>>>>> [c0000000eec539a0] [c0000000000dfa88] .swap_readpage+0x94/0xb4 >>>>>> [c0000000eec53a20] [c0000000000e02b8] >>>>>> .read_swap_cache_async+0x110/0x160 >>>>>> [c0000000eec53ae0] [c0000000000e035c] .swapin_readahead+0x54/0xd4 >>>>>> [c0000000eec53ba0] [c0000000000d47e8] .handle_mm_fault+0x520/0x9d8 >>>>>> [c0000000eec53c80] [c0000000004b5054] .do_page_fault+0x440/0x624 >>>>>> [c0000000eec53e30] [c0000000000053fc] handle_page_fault+0x20/0x5c >>> >>> >>> Kamalesh, >>> >>> I'm wondering whether this is not related to a problem found in LTP >>> after msgmni had been changed from 16 to a computed value. >>> >>> Does your ltp test suite contain the patch I had sent in thread >>> http://lkml.org/lkml/2008/2/21/123? >>> >>> If you want to have a look at it, discussion started in thread >>> http://lkml.org/lkml/2008/2/16/6 >>> >>> Regards, >>> Nadia >>> >> >> Hi Nadia, >> Thanks, I am using the LTP-20080430 and tried it with the >> 2.6.26-rc2-git4 kernel. >> but the call trace remains the same. >> > > Ok, I'll try to find a ppc machine: could not reproduce it on my Xeon > machine (but actually it was with a 2.6.26-rc1 kernel). > > Regards, > Nadia > Kamalesh, I could find a ppc victim, and ... could not reproduce the problem. Attached is the output I got. Regards, Nadia [-- Attachment #2: msgctl08.output --] [-- Type: text/plain, Size: 3487 bytes --] [root@machette ltp-full-20080430]# cat runtest/ipc.fail msgctl08 msgctl08 [root@machette ltp-full-20080430]# ./runltp -p -l result.03.log -f ipc.fail INFO: creating /home/lkernel/src/ltp-full-20080430/output directory INFO: creating /home/lkernel/src/ltp-full-20080430/results directory If some fields are empty or look unusual you may have an old version. Compare to the current minimal requirements in Documentation/Changes. Red Hat Enterprise Linux Server release 5.1 (Tikanga) Linux machette 2.6.26-rc1 #1 SMP PREEMPT Mon May 19 12:00:48 CEST 2008 ppc64 ppc 64 ppc64 GNU/Linux Gnu C gcc (GCC) 4.1.2 20070626 (Red Hat 4.1.2-14) Gnu make 3.81 util-linux pre7) mount pre7) modutils 3.3-pre2 e2fsprogs 1.39 PPP 2.4.4 isdn4k-utils 3.9 Linux C Library > libc.2.5 Dynamic linker (ldd) 2.5 Procps 3.2.7 Net-tools 1.60 Kbd 1.12 Sh-utils 5.97 Modules Loaded nfsd exportfs lockd sunrpc ehea sg ipr libata sd_mod scsi _mod ext3 jbd free reports: total used free shared buffers cached Mem: 7717720 202796 7514924 0 16236 83800 -/+ buffers/cache: 102760 7614960 Swap: 0 0 0 /proc/cpuinfo processor : 0 cpu : POWER6 (raw), altivec supported clock : 4005.000000MHz revision : 3.1 (pvr 003e 0301) processor : 1 cpu : POWER6 (raw), altivec supported clock : 4005.000000MHz revision : 3.1 (pvr 003e 0301) processor : 2 cpu : POWER6 (raw), altivec supported clock : 4005.000000MHz revision : 3.1 (pvr 003e 0301) processor : 3 cpu : POWER6 (raw), altivec supported clock : 4005.000000MHz revision : 3.1 (pvr 003e 0301) processor : 4 cpu : POWER6 (raw), altivec supported clock : 4005.000000MHz revision : 3.1 (pvr 003e 0301) processor : 5 cpu : POWER6 (raw), altivec supported clock : 4005.000000MHz revision : 3.1 (pvr 003e 0301) processor : 6 cpu : POWER6 (raw), altivec supported clock : 4005.000000MHz revision : 3.1 (pvr 003e 0301) processor : 7 cpu : POWER6 (raw), altivec supported clock : 4005.000000MHz revision : 3.1 (pvr 003e 0301) timebase : 512000000 platform : pSeries model : IBM,7998-61X machine : CHRP IBM,7998-61X <<<test_start>>> tag=msgctl08 stime=1211192505 cmdline="msgctl08" contacts="" analysis=exit initiation_status="ok" <<<test_output>>> incrementing stop msgctl08 1 PASS : msgctl08 ran successfully! <<<execution_status>>> duration=2 termination_type=exited termination_id=0 corefile=no cutime=148 cstime=357 <<<test_end>>> INFO: pan reported all tests PASS LTP Version: LTP-20080430 ###############################################################" Done executing testcases." log result(s) is in /home/lkernel/src/ltp-full-20080430/results directory" output/failed result(s) in /home/lkernel/src/ltp-full-20080430/output directory [If you did not provide Absolute Path] LTP Version: LTP-20080430 ###############################################################" ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2008-05-19 10:32 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2008-05-07 13:22 [BUG] 2.6.26-rc1-git4 - task blocked on powerpc for more than 120 seconds Kamalesh Babulal 2008-05-07 13:40 ` Adrian Bunk 2008-05-08 3:59 ` Kamalesh Babulal 2008-05-15 11:30 ` Nadia Derbey 2008-05-16 14:26 ` Kamalesh Babulal 2008-05-19 8:23 ` Nadia Derbey 2008-05-19 10:14 ` Kamalesh Babulal 2008-05-19 10:31 ` Nadia Derbey
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).