* [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests @ 2021-06-21 6:32 Sachin Sant 2021-06-21 9:39 ` Odin Ugedal 0 siblings, 1 reply; 13+ messages in thread From: Sachin Sant @ 2021-06-21 6:32 UTC (permalink / raw) To: linux-kernel; +Cc: peterz, odin, linuxppc-dev While running LTP tests (cfs_bandwidth01) against 5.13.0-rc7 kernel on a powerpc box following warning is seen [ 6611.331827] ------------[ cut here ]------------ [ 6611.331855] rq->tmp_alone_branch != &rq->leaf_cfs_rq_list [ 6611.331862] WARNING: CPU: 8 PID: 0 at kernel/sched/fair.c:401 unthrottle_cfs_rq+0x4cc/0x590 [ 6611.331883] Modules linked in: nfsv3 nfs_acl nfs lockd grace fscache netfs tun brd overlay vfat fat btrfs blake2b_generic xor zstd_compress raid6_pq xfs loop sctp ip6_udp_tunnel udp_tunnel libcrc32c dm_mod bonding rfkill sunrpc pseries_rng xts vmx_crypto sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi sg ibmvscsi ibmveth scsi_transport_srp fuse [last unloaded: init_module] [ 6611.331957] CPU: 8 PID: 0 Comm: swapper/8 Tainted: G OE 5.13.0-rc6-gcba5e97280f5 #1 [ 6611.331968] NIP: c0000000001b7aac LR: c0000000001b7aa8 CTR: c000000000722d30 [ 6611.331976] REGS: c00000000274f3a0 TRAP: 0700 Tainted: G OE (5.13.0-rc6-gcba5e97280f5) [ 6611.331985] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE> CR: 48000224 XER: 00000005 [ 6611.332002] CFAR: c00000000014ca20 IRQMASK: 1 [ 6611.332002] GPR00: c0000000001b7aa8 c00000000274f640 c000000001abaf00 000000000000002d [ 6611.332002] GPR04: 00000000ffff7fff c00000000274f300 0000000000000027 c000000efdb07e08 [ 6611.332002] GPR08: 0000000000000023 0000000000000001 0000000000000027 c000000001976680 [ 6611.332002] GPR12: 0000000000000000 c000000effc0be80 c000000ef07b3f90 000000001eefe200 [ 6611.332002] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 6611.332002] GPR20: 0000000000000001 c000000000fa6c08 c000000000fa6030 0000000000000001 [ 6611.332002] GPR24: 0000000000000000 0000000000000000 c000000efde12380 0000000000000001 [ 6611.332002] GPR28: 0000000000000001 0000000000000000 c000000efde12400 0000000000000000 [ 6611.332094] NIP [c0000000001b7aac] unthrottle_cfs_rq+0x4cc/0x590 [ 6611.332104] LR [c0000000001b7aa8] unthrottle_cfs_rq+0x4c8/0x590 [ 6611.332113] Call Trace: [ 6611.332116] [c00000000274f640] [c0000000001b7aa8] unthrottle_cfs_rq+0x4c8/0x590 (unreliable) [ 6611.332128] [c00000000274f6e0] [c0000000001b7e38] distribute_cfs_runtime+0x1d8/0x280 [ 6611.332139] [c00000000274f7b0] [c0000000001b81d0] sched_cfs_period_timer+0x140/0x330 [ 6611.332149] [c00000000274f870] [c00000000022a03c] __hrtimer_run_queues+0x17c/0x380 [ 6611.332158] [c00000000274f8f0] [c00000000022ac68] hrtimer_interrupt+0x128/0x2f0 [ 6611.332168] [c00000000274f9a0] [c00000000002940c] timer_interrupt+0x13c/0x370 [ 6611.332179] [c00000000274fa00] [c000000000009c04] decrementer_common_virt+0x1a4/0x1b0 [ 6611.332189] --- interrupt: 900 at plpar_hcall_norets_notrace+0x18/0x24 [ 6611.332199] NIP: c0000000000f6af8 LR: c000000000a05f68 CTR: 0000000000000000 [ 6611.332206] REGS: c00000000274fa70 TRAP: 0900 Tainted: G OE (5.13.0-rc6-gcba5e97280f5) [ 6611.332214] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 28000224 XER: 00000000 [ 6611.332234] CFAR: 0000000000000c00 IRQMASK: 0 [ 6611.332234] GPR00: 0000000000000000 c00000000274fd10 c000000001abaf00 0000000000000000 [ 6611.332234] GPR04: 00000000000000c0 0000000000000080 0001a91c68b80fa1 00000000000003dc [ 6611.332234] GPR08: 000000000001f400 0000000000000001 0000000000000000 0000000000000000 [ 6611.332234] GPR12: 0000000000000000 c000000effc0be80 c000000ef07b3f90 000000001eefe200 [ 6611.332234] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 6611.332234] GPR20: 0000000000000001 0000000000000002 0000000000000010 c0000000019fe2f8 [ 6611.332234] GPR24: 0000000000000001 00000603517d757e 0000000000000000 0000000000000000 [ 6611.332234] GPR28: 0000000000000001 0000000000000000 c000000001231f90 c000000001231f98 [ 6611.332323] NIP [c0000000000f6af8] plpar_hcall_norets_notrace+0x18/0x24 [ 6611.332332] LR [c000000000a05f68] check_and_cede_processor+0x48/0x60 [ 6611.332340] --- interrupt: 900 [ 6611.332345] [c00000000274fd10] [c000000efdb92380] 0xc000000efdb92380 (unreliable) [ 6611.332355] [c00000000274fd70] [c000000000a063bc] dedicated_cede_loop+0x9c/0x1b0 [ 6611.332364] [c00000000274fdc0] [c000000000a02b04] cpuidle_enter_state+0x2e4/0x4e0 [ 6611.332375] [c00000000274fe20] [c000000000a02da0] cpuidle_enter+0x50/0x70 [ 6611.332385] [c00000000274fe60] [c0000000001a883c] call_cpuidle+0x4c/0x80 [ 6611.332393] [c00000000274fe80] [c0000000001a8ee0] do_idle+0x380/0x3e0 [ 6611.332402] [c00000000274ff00] [c0000000001a91bc] cpu_startup_entry+0x3c/0x40 [ 6611.332411] [c00000000274ff30] [c000000000063ff8] start_secondary+0x298/0x2b0 [ 6611.332421] [c00000000274ff90] [c00000000000c754] start_secondary_prolog+0x10/0x14 [ 6611.332430] Instruction dump: [ 6611.332435] 4bfffc44 3d22fff6 8929f328 2f890000 409efea4 39200001 3d42fff6 3c62ff4f [ 6611.332451] 3863bcd8 992af328 4bf94f15 60000000 <0fe00000> 4bfffe80 7f6407b4 7f43d378 [ 6611.332466] ---[ end trace 1346f865cd1cae91 ]— 5.13.0-rc6 was good. Bisect points to following patch commit a7b359fc6a37 sched/fair: Correctly insert cfs_rq's to list on unthrottle The test runs to completion(without this warning) if the patch is reverted. Thanks -Sachin ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests 2021-06-21 6:32 [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests Sachin Sant @ 2021-06-21 9:39 ` Odin Ugedal 2021-06-21 9:50 ` Vincent Guittot 0 siblings, 1 reply; 13+ messages in thread From: Odin Ugedal @ 2021-06-21 9:39 UTC (permalink / raw) To: Sachin Sant Cc: Peter Zijlstra, Odin Ugedal, linuxppc-dev, open list, Vincent Guittot man. 21. jun. 2021 kl. 08:33 skrev Sachin Sant <sachinp@linux.vnet.ibm.com>: > > While running LTP tests (cfs_bandwidth01) against 5.13.0-rc7 kernel on a powerpc box > following warning is seen > > [ 6611.331827] ------------[ cut here ]------------ > [ 6611.331855] rq->tmp_alone_branch != &rq->leaf_cfs_rq_list > [ 6611.331862] WARNING: CPU: 8 PID: 0 at kernel/sched/fair.c:401 unthrottle_cfs_rq+0x4cc/0x590 > [ 6611.331883] Modules linked in: nfsv3 nfs_acl nfs lockd grace fscache netfs tun brd overlay vfat fat btrfs blake2b_generic xor zstd_compress raid6_pq xfs loop sctp ip6_udp_tunnel udp_tunnel libcrc32c dm_mod bonding rfkill sunrpc pseries_rng xts vmx_crypto sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi sg ibmvscsi ibmveth scsi_transport_srp fuse [last unloaded: init_module] > [ 6611.331957] CPU: 8 PID: 0 Comm: swapper/8 Tainted: G OE 5.13.0-rc6-gcba5e97280f5 #1 > [ 6611.331968] NIP: c0000000001b7aac LR: c0000000001b7aa8 CTR: c000000000722d30 > [ 6611.331976] REGS: c00000000274f3a0 TRAP: 0700 Tainted: G OE (5.13.0-rc6-gcba5e97280f5) > [ 6611.331985] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE> CR: 48000224 XER: 00000005 > [ 6611.332002] CFAR: c00000000014ca20 IRQMASK: 1 > [ 6611.332002] GPR00: c0000000001b7aa8 c00000000274f640 c000000001abaf00 000000000000002d > [ 6611.332002] GPR04: 00000000ffff7fff c00000000274f300 0000000000000027 c000000efdb07e08 > [ 6611.332002] GPR08: 0000000000000023 0000000000000001 0000000000000027 c000000001976680 > [ 6611.332002] GPR12: 0000000000000000 c000000effc0be80 c000000ef07b3f90 000000001eefe200 > [ 6611.332002] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > [ 6611.332002] GPR20: 0000000000000001 c000000000fa6c08 c000000000fa6030 0000000000000001 > [ 6611.332002] GPR24: 0000000000000000 0000000000000000 c000000efde12380 0000000000000001 > [ 6611.332002] GPR28: 0000000000000001 0000000000000000 c000000efde12400 0000000000000000 > [ 6611.332094] NIP [c0000000001b7aac] unthrottle_cfs_rq+0x4cc/0x590 > [ 6611.332104] LR [c0000000001b7aa8] unthrottle_cfs_rq+0x4c8/0x590 > [ 6611.332113] Call Trace: > [ 6611.332116] [c00000000274f640] [c0000000001b7aa8] unthrottle_cfs_rq+0x4c8/0x590 (unreliable) > [ 6611.332128] [c00000000274f6e0] [c0000000001b7e38] distribute_cfs_runtime+0x1d8/0x280 > [ 6611.332139] [c00000000274f7b0] [c0000000001b81d0] sched_cfs_period_timer+0x140/0x330 > [ 6611.332149] [c00000000274f870] [c00000000022a03c] __hrtimer_run_queues+0x17c/0x380 > [ 6611.332158] [c00000000274f8f0] [c00000000022ac68] hrtimer_interrupt+0x128/0x2f0 > [ 6611.332168] [c00000000274f9a0] [c00000000002940c] timer_interrupt+0x13c/0x370 > [ 6611.332179] [c00000000274fa00] [c000000000009c04] decrementer_common_virt+0x1a4/0x1b0 > [ 6611.332189] --- interrupt: 900 at plpar_hcall_norets_notrace+0x18/0x24 > [ 6611.332199] NIP: c0000000000f6af8 LR: c000000000a05f68 CTR: 0000000000000000 > [ 6611.332206] REGS: c00000000274fa70 TRAP: 0900 Tainted: G OE (5.13.0-rc6-gcba5e97280f5) > [ 6611.332214] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 28000224 XER: 00000000 > [ 6611.332234] CFAR: 0000000000000c00 IRQMASK: 0 > [ 6611.332234] GPR00: 0000000000000000 c00000000274fd10 c000000001abaf00 0000000000000000 > [ 6611.332234] GPR04: 00000000000000c0 0000000000000080 0001a91c68b80fa1 00000000000003dc > [ 6611.332234] GPR08: 000000000001f400 0000000000000001 0000000000000000 0000000000000000 > [ 6611.332234] GPR12: 0000000000000000 c000000effc0be80 c000000ef07b3f90 000000001eefe200 > [ 6611.332234] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > [ 6611.332234] GPR20: 0000000000000001 0000000000000002 0000000000000010 c0000000019fe2f8 > [ 6611.332234] GPR24: 0000000000000001 00000603517d757e 0000000000000000 0000000000000000 > [ 6611.332234] GPR28: 0000000000000001 0000000000000000 c000000001231f90 c000000001231f98 > [ 6611.332323] NIP [c0000000000f6af8] plpar_hcall_norets_notrace+0x18/0x24 > [ 6611.332332] LR [c000000000a05f68] check_and_cede_processor+0x48/0x60 > [ 6611.332340] --- interrupt: 900 > [ 6611.332345] [c00000000274fd10] [c000000efdb92380] 0xc000000efdb92380 (unreliable) > [ 6611.332355] [c00000000274fd70] [c000000000a063bc] dedicated_cede_loop+0x9c/0x1b0 > [ 6611.332364] [c00000000274fdc0] [c000000000a02b04] cpuidle_enter_state+0x2e4/0x4e0 > [ 6611.332375] [c00000000274fe20] [c000000000a02da0] cpuidle_enter+0x50/0x70 > [ 6611.332385] [c00000000274fe60] [c0000000001a883c] call_cpuidle+0x4c/0x80 > [ 6611.332393] [c00000000274fe80] [c0000000001a8ee0] do_idle+0x380/0x3e0 > [ 6611.332402] [c00000000274ff00] [c0000000001a91bc] cpu_startup_entry+0x3c/0x40 > [ 6611.332411] [c00000000274ff30] [c000000000063ff8] start_secondary+0x298/0x2b0 > [ 6611.332421] [c00000000274ff90] [c00000000000c754] start_secondary_prolog+0x10/0x14 > [ 6611.332430] Instruction dump: > [ 6611.332435] 4bfffc44 3d22fff6 8929f328 2f890000 409efea4 39200001 3d42fff6 3c62ff4f > [ 6611.332451] 3863bcd8 992af328 4bf94f15 60000000 <0fe00000> 4bfffe80 7f6407b4 7f43d378 > [ 6611.332466] ---[ end trace 1346f865cd1cae91 ]— > > 5.13.0-rc6 was good. Bisect points to following patch > > commit a7b359fc6a37 > sched/fair: Correctly insert cfs_rq's to list on unthrottle > > The test runs to completion(without this warning) if the patch is reverted. > > Thanks > -Sachin > Hi, Thanks for the report! I have a theory about what is possibly causing this, so I will try to reproduce it and see if my assumptions are correct. Odin ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests 2021-06-21 9:39 ` Odin Ugedal @ 2021-06-21 9:50 ` Vincent Guittot 2021-06-21 9:54 ` Odin Ugedal 0 siblings, 1 reply; 13+ messages in thread From: Vincent Guittot @ 2021-06-21 9:50 UTC (permalink / raw) To: Odin Ugedal; +Cc: Sachin Sant, Peter Zijlstra, linuxppc-dev, open list On Mon, 21 Jun 2021 at 11:39, Odin Ugedal <odin@uged.al> wrote: > > man. 21. jun. 2021 kl. 08:33 skrev Sachin Sant <sachinp@linux.vnet.ibm.com>: > > > > While running LTP tests (cfs_bandwidth01) against 5.13.0-rc7 kernel on a powerpc box > > following warning is seen > > > > [ 6611.331827] ------------[ cut here ]------------ > > [ 6611.331855] rq->tmp_alone_branch != &rq->leaf_cfs_rq_list > > [ 6611.331862] WARNING: CPU: 8 PID: 0 at kernel/sched/fair.c:401 unthrottle_cfs_rq+0x4cc/0x590 > > [ 6611.331883] Modules linked in: nfsv3 nfs_acl nfs lockd grace fscache netfs tun brd overlay vfat fat btrfs blake2b_generic xor zstd_compress raid6_pq xfs loop sctp ip6_udp_tunnel udp_tunnel libcrc32c dm_mod bonding rfkill sunrpc pseries_rng xts vmx_crypto sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod t10_pi sg ibmvscsi ibmveth scsi_transport_srp fuse [last unloaded: init_module] > > [ 6611.331957] CPU: 8 PID: 0 Comm: swapper/8 Tainted: G OE 5.13.0-rc6-gcba5e97280f5 #1 > > [ 6611.331968] NIP: c0000000001b7aac LR: c0000000001b7aa8 CTR: c000000000722d30 > > [ 6611.331976] REGS: c00000000274f3a0 TRAP: 0700 Tainted: G OE (5.13.0-rc6-gcba5e97280f5) > > [ 6611.331985] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE> CR: 48000224 XER: 00000005 > > [ 6611.332002] CFAR: c00000000014ca20 IRQMASK: 1 > > [ 6611.332002] GPR00: c0000000001b7aa8 c00000000274f640 c000000001abaf00 000000000000002d > > [ 6611.332002] GPR04: 00000000ffff7fff c00000000274f300 0000000000000027 c000000efdb07e08 > > [ 6611.332002] GPR08: 0000000000000023 0000000000000001 0000000000000027 c000000001976680 > > [ 6611.332002] GPR12: 0000000000000000 c000000effc0be80 c000000ef07b3f90 000000001eefe200 > > [ 6611.332002] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > > [ 6611.332002] GPR20: 0000000000000001 c000000000fa6c08 c000000000fa6030 0000000000000001 > > [ 6611.332002] GPR24: 0000000000000000 0000000000000000 c000000efde12380 0000000000000001 > > [ 6611.332002] GPR28: 0000000000000001 0000000000000000 c000000efde12400 0000000000000000 > > [ 6611.332094] NIP [c0000000001b7aac] unthrottle_cfs_rq+0x4cc/0x590 > > [ 6611.332104] LR [c0000000001b7aa8] unthrottle_cfs_rq+0x4c8/0x590 > > [ 6611.332113] Call Trace: > > [ 6611.332116] [c00000000274f640] [c0000000001b7aa8] unthrottle_cfs_rq+0x4c8/0x590 (unreliable) > > [ 6611.332128] [c00000000274f6e0] [c0000000001b7e38] distribute_cfs_runtime+0x1d8/0x280 > > [ 6611.332139] [c00000000274f7b0] [c0000000001b81d0] sched_cfs_period_timer+0x140/0x330 > > [ 6611.332149] [c00000000274f870] [c00000000022a03c] __hrtimer_run_queues+0x17c/0x380 > > [ 6611.332158] [c00000000274f8f0] [c00000000022ac68] hrtimer_interrupt+0x128/0x2f0 > > [ 6611.332168] [c00000000274f9a0] [c00000000002940c] timer_interrupt+0x13c/0x370 > > [ 6611.332179] [c00000000274fa00] [c000000000009c04] decrementer_common_virt+0x1a4/0x1b0 > > [ 6611.332189] --- interrupt: 900 at plpar_hcall_norets_notrace+0x18/0x24 > > [ 6611.332199] NIP: c0000000000f6af8 LR: c000000000a05f68 CTR: 0000000000000000 > > [ 6611.332206] REGS: c00000000274fa70 TRAP: 0900 Tainted: G OE (5.13.0-rc6-gcba5e97280f5) > > [ 6611.332214] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 28000224 XER: 00000000 > > [ 6611.332234] CFAR: 0000000000000c00 IRQMASK: 0 > > [ 6611.332234] GPR00: 0000000000000000 c00000000274fd10 c000000001abaf00 0000000000000000 > > [ 6611.332234] GPR04: 00000000000000c0 0000000000000080 0001a91c68b80fa1 00000000000003dc > > [ 6611.332234] GPR08: 000000000001f400 0000000000000001 0000000000000000 0000000000000000 > > [ 6611.332234] GPR12: 0000000000000000 c000000effc0be80 c000000ef07b3f90 000000001eefe200 > > [ 6611.332234] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > > [ 6611.332234] GPR20: 0000000000000001 0000000000000002 0000000000000010 c0000000019fe2f8 > > [ 6611.332234] GPR24: 0000000000000001 00000603517d757e 0000000000000000 0000000000000000 > > [ 6611.332234] GPR28: 0000000000000001 0000000000000000 c000000001231f90 c000000001231f98 > > [ 6611.332323] NIP [c0000000000f6af8] plpar_hcall_norets_notrace+0x18/0x24 > > [ 6611.332332] LR [c000000000a05f68] check_and_cede_processor+0x48/0x60 > > [ 6611.332340] --- interrupt: 900 > > [ 6611.332345] [c00000000274fd10] [c000000efdb92380] 0xc000000efdb92380 (unreliable) > > [ 6611.332355] [c00000000274fd70] [c000000000a063bc] dedicated_cede_loop+0x9c/0x1b0 > > [ 6611.332364] [c00000000274fdc0] [c000000000a02b04] cpuidle_enter_state+0x2e4/0x4e0 > > [ 6611.332375] [c00000000274fe20] [c000000000a02da0] cpuidle_enter+0x50/0x70 > > [ 6611.332385] [c00000000274fe60] [c0000000001a883c] call_cpuidle+0x4c/0x80 > > [ 6611.332393] [c00000000274fe80] [c0000000001a8ee0] do_idle+0x380/0x3e0 > > [ 6611.332402] [c00000000274ff00] [c0000000001a91bc] cpu_startup_entry+0x3c/0x40 > > [ 6611.332411] [c00000000274ff30] [c000000000063ff8] start_secondary+0x298/0x2b0 > > [ 6611.332421] [c00000000274ff90] [c00000000000c754] start_secondary_prolog+0x10/0x14 > > [ 6611.332430] Instruction dump: > > [ 6611.332435] 4bfffc44 3d22fff6 8929f328 2f890000 409efea4 39200001 3d42fff6 3c62ff4f > > [ 6611.332451] 3863bcd8 992af328 4bf94f15 60000000 <0fe00000> 4bfffe80 7f6407b4 7f43d378 > > [ 6611.332466] ---[ end trace 1346f865cd1cae91 ]— > > > > 5.13.0-rc6 was good. Bisect points to following patch > > > > commit a7b359fc6a37 > > sched/fair: Correctly insert cfs_rq's to list on unthrottle > > > > The test runs to completion(without this warning) if the patch is reverted. > > > > Thanks > > -Sachin > > > > Hi, > > Thanks for the report! I have a theory about what is possibly causing > this, so I will try to reproduce it and see if my assumptions are > correct. This means that a child's load was not null and it was inserted whereas parent's load was null. This should not happen unless the propagation failed somewhere > > > Odin ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests 2021-06-21 9:50 ` Vincent Guittot @ 2021-06-21 9:54 ` Odin Ugedal 2021-06-21 10:57 ` Sachin Sant 0 siblings, 1 reply; 13+ messages in thread From: Odin Ugedal @ 2021-06-21 9:54 UTC (permalink / raw) To: Vincent Guittot Cc: Sachin Sant, linuxppc-dev, Peter Zijlstra, Odin Ugedal, open list man. 21. jun. 2021 kl. 11:50 skrev Vincent Guittot <vincent.guittot@linaro.org>: > This means that a child's load was not null and it was inserted > whereas parent's load was null. This should not happen unless the > propagation failed somewhere My initial thought is that the patch below will fix it, if that is the issue (that a leaf is inserted, but the propagation is not "completed" in unthrottle). Might that be the case? Still working on reproducing the issue tho. diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index bfaa6e1f6067..015c5a5c1a4d 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4930,12 +4930,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) if (cfs_rq_throttled(cfs_rq)) goto unthrottle_throttle; - /* - * One parent has been throttled and cfs_rq removed from the - * list. Add it back to not break the leaf list. - */ - if (throttled_hierarchy(cfs_rq)) - list_add_leaf_cfs_rq(cfs_rq); + list_add_leaf_cfs_rq(cfs_rq); } /* At this point se is NULL and we are at root level*/ ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests 2021-06-21 9:54 ` Odin Ugedal @ 2021-06-21 10:57 ` Sachin Sant 2021-06-21 11:04 ` Odin Ugedal 0 siblings, 1 reply; 13+ messages in thread From: Sachin Sant @ 2021-06-21 10:57 UTC (permalink / raw) To: Odin Ugedal; +Cc: Peter Zijlstra, linuxppc-dev, Vincent Guittot, open list [-- Attachment #1: Type: text/plain, Size: 1475 bytes --] > On 21-Jun-2021, at 3:24 PM, Odin Ugedal <odin@uged.al> wrote: > > man. 21. jun. 2021 kl. 11:50 skrev Vincent Guittot <vincent.guittot@linaro.org>: >> This means that a child's load was not null and it was inserted >> whereas parent's load was null. This should not happen unless the >> propagation failed somewhere > > My initial thought is that the patch below will fix it, if that is the > issue (that a leaf is inserted, but the propagation is not "completed" > in unthrottle). Might that be the case? Still working on reproducing > the issue tho. > Unfortunately this does not help. I can still recreate the failure. Have attached the o/p from test run. Thanks -Sachin > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c > index bfaa6e1f6067..015c5a5c1a4d 100644 > --- a/kernel/sched/fair.c > +++ b/kernel/sched/fair.c > @@ -4930,12 +4930,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) > if (cfs_rq_throttled(cfs_rq)) > goto unthrottle_throttle; > > - /* > - * One parent has been throttled and cfs_rq removed from the > - * list. Add it back to not break the leaf list. > - */ > - if (throttled_hierarchy(cfs_rq)) > - list_add_leaf_cfs_rq(cfs_rq); > + list_add_leaf_cfs_rq(cfs_rq); > } > > /* At this point se is NULL and we are at root level*/ [-- Attachment #2: run.txt --] [-- Type: text/plain, Size: 5966 bytes --] # cd /opt/ltp/testcases/bin # ./cfs_bandwidth01 -i 5 tst_test.c:1313: TINFO: Timeout per run is 0h 05m 00s tst_buffers.c:55: TINFO: Test is using guarded buffers cfs_bandwidth01.c:49: TINFO: Set 'worker1/cpu.max' = '3000 10000' cfs_bandwidth01.c:49: TINFO: Set 'worker2/cpu.max' = '2000 10000' cfs_bandwidth01.c:49: TINFO: Set 'worker3/cpu.max' = '3000 10000' cfs_bandwidth01.c:113: TPASS: Scheduled bandwidth constrained workers cfs_bandwidth01.c:49: TINFO: Set 'level2/cpu.max' = '5000 10000' cfs_bandwidth01.c:125: TPASS: Workers exited cfs_bandwidth01.c:113: TPASS: Scheduled bandwidth constrained workers [ 48.343143] ------------[ cut here ]------------ [ 48.343164] rq->tmp_alone_branch != &rq->leaf_cfs_rq_list [ 48.343172] WARNING: CPU: 24 PID: 4405 at kernel/sched/fair.c:401 unthrottle_cfs_rq+0x49c/0x560 [ 48.343196] Modules linked in: nf_tables nfnetlink tun bridge stp llc rfkill sunrpc pseries_rng xts vmx_crypto uio_pdrv_genirq uio sch_fq_codel ip_tables xfs libcrc32c sr_mod sd_mod cdrom t10_pi sg ibmvscsi ibmveth scsi_transport_srp dm_mirror dm_region_hash dm_log dm_mod fuse [ 48.343251] CPU: 24 PID: 4405 Comm: cfs_bandwidth01 Not tainted 5.13.0-rc7-dirty #4 [ 48.343261] NIP: c0000000001b88fc LR: c0000000001b88f8 CTR: c000000000723d10 [ 48.343269] REGS: c00000000fb13780 TRAP: 0700 Not tainted (5.13.0-rc7-dirty) [ 48.343278] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE> CR: 48044224 XER: 00000005 [ 48.343295] CFAR: c00000000014d8a0 IRQMASK: 1 [ 48.343295] GPR00: c0000000001b88f8 c00000000fb13a20 c0000000029ab400 000000000000002d [ 48.343295] GPR04: 00000000fffeffff c00000000fb136e0 0000000000000027 c00000154f817e08 [ 48.343295] GPR08: 0000000000000023 0000000000000001 0000000000000027 c00000167f1d7fe8 [ 48.343295] GPR12: 0000000000004000 c00000154ffdc680 0000000000000000 0000000000000000 [ 48.343295] GPR16: c000000000fa6660 0000000000000001 0000000000000000 c0000000024e1cd8 [ 48.343295] GPR20: 0000000000000000 c00000000290a69a 0000000000000000 c0000000024e1cc0 [ 48.343295] GPR24: 0000000000000000 c0000000029f2140 c00000154f762380 0000000000000001 [ 48.343295] GPR28: 0000000000000001 0000000000000000 c00000154f762400 0000000000000000 [ 48.343388] NIP [c0000000001b88fc] unthrottle_cfs_rq+0x49c/0x560 [ 48.343397] LR [c0000000001b88f8] unthrottle_cfs_rq+0x498/0x560 [ 48.343406] Call Trace: [ 48.343410] [c00000000fb13a20] [c0000000001b88f8] unthrottle_cfs_rq+0x498/0x560 (unreliable) [ 48.343422] [c00000000fb13ac0] [c00000000019edb8] tg_set_cfs_bandwidth+0x2c8/0x470 [ 48.343433] [c00000000fb13bc0] [c000000000263874] cgroup_file_write+0x164/0x210 [ 48.343444] [c00000000fb13c20] [c00000000058cfac] kernfs_fop_write_iter+0x1cc/0x280 [ 48.343455] [c00000000fb13c70] [c00000000047024c] new_sync_write+0x14c/0x1d0 [ 48.343467] [c00000000fb13d10] [c000000000473844] vfs_write+0x224/0x330 [ 48.343476] [c00000000fb13d60] [c000000000473b2c] ksys_write+0x7c/0x140 [ 48.343485] [c00000000fb13db0] [c000000000030fb0] system_call_exception+0x150/0x2d0 [ 48.343495] [c00000000fb13e10] [c00000000000d45c] system_call_common+0xec/0x278 [ 48.343504] --- interrupt: c00 at 0x7fffaa67bd74 [ 48.343511] NIP: 00007fffaa67bd74 LR: 00007fffaa5f34c4 CTR: 0000000000000000 [ 48.343519] REGS: c00000000fb13e80 TRAP: 0c00 Not tainted (5.13.0-rc7-dirty) [ 48.343527] MSR: 800000000280f033 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 28002282 XER: 00000000 [ 48.343548] IRQMASK: 0 [ 48.343548] GPR00: 0000000000000004 00007fffcb534d60 00007fffaa777100 0000000000000010 [ 48.343548] GPR04: 00000000415623d0 0000000000000005 0000000000000010 00007fffcb534df8 [ 48.343548] GPR08: 0000000010028618 0000000000000000 0000000000000000 0000000000000000 [ 48.343548] GPR12: 0000000000000000 00007fffaa81a310 0000000000000000 0000000000000000 [ 48.343548] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 48.343548] GPR20: 0000000000000000 0000000000000000 0000000000000002 0000000000000000 [ 48.343548] GPR24: 0000000000000000 000000000000002b 0000000000000005 00000000415623d0 [ 48.343548] GPR28: 0000000000000005 00007fffcb534eb0 00000000415623d0 0000000000000005 [ 48.343634] NIP [00007fffaa67bd74] 0x7fffaa67bd74 [ 48.343640] LR [00007fffaa5f34c4] 0x7fffaa5f34c4 [ 48.343646] --- interrupt: c00 [ 48.343651] Instruction dump: [ 48.343656] 4bfffc74 3d22fff6 8929f2a9 2f890000 409efed4 39200001 3d42fff6 3c62fe60 [ 48.343672] 3863be08 992af2a9 4bf94f45 60000000 <0fe00000> 4bfffeb0 7f6407b4 7f43d378 [ 48.343687] ---[ end trace 61db91af8340603f ]--- cfs_bandwidth01.c:49: TINFO: Set 'level2/cpu.max' = '5000 10000' cfs_bandwidth01.c:125: TPASS: Workers exited cfs_bandwidth01.c:113: TPASS: Scheduled bandwidth constrained workers cfs_bandwidth01.c:49: TINFO: Set 'level2/cpu.max' = '5000 10000' cfs_bandwidth01.c:125: TPASS: Workers exited cfs_bandwidth01.c:113: TPASS: Scheduled bandwidth constrained workers cfs_bandwidth01.c:49: TINFO: Set 'level2/cpu.max' = '5000 10000' cfs_bandwidth01.c:125: TPASS: Workers exited cfs_bandwidth01.c:113: TPASS: Scheduled bandwidth constrained workers cfs_bandwidth01.c:49: TINFO: Set 'level2/cpu.max' = '5000 10000' cfs_bandwidth01.c:125: TPASS: Workers exited tst_test.c:1349: TFAIL: Kernel is now tainted. HINT: You _MAY_ be missing kernel fixes, see: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=39f23ce07b93 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b34cb07dde7c https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fe61468b2cbc https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5ab297bab984 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6d4d22468dae Summary: passed 10 failed 1 broken 0 skipped 0 warnings 0 ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests 2021-06-21 10:57 ` Sachin Sant @ 2021-06-21 11:04 ` Odin Ugedal 2021-06-21 12:42 ` Odin Ugedal 0 siblings, 1 reply; 13+ messages in thread From: Odin Ugedal @ 2021-06-21 11:04 UTC (permalink / raw) To: Sachin Sant Cc: Peter Zijlstra, Vincent Guittot, linuxppc-dev, Odin Ugedal, open list man. 21. jun. 2021 kl. 12:57 skrev Sachin Sant <sachinp@linux.vnet.ibm.com>: > > Unfortunately this does not help. I can still recreate the failure. > > Have attached the o/p from test run. > > Thanks > -Sachin Yes, thanks! I am able to reproduce it locally now, so will keep looking to see if I find the cause. Thanks! Odin ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests 2021-06-21 11:04 ` Odin Ugedal @ 2021-06-21 12:42 ` Odin Ugedal 2021-06-21 16:22 ` Vincent Guittot 0 siblings, 1 reply; 13+ messages in thread From: Odin Ugedal @ 2021-06-21 12:42 UTC (permalink / raw) To: Odin Ugedal Cc: Sachin Sant, linuxppc-dev, Peter Zijlstra, Vincent Guittot, open list Hi, Did some more research, and it looks like this is what happens: $ tree /sys/fs/cgroup/ltp/ -d --charset=ascii /sys/fs/cgroup/ltp/ |-- drain `-- test-6851 `-- level2 |-- level3a | |-- worker1 | `-- worker2 `-- level3b `-- worker3 Timeline (ish): - worker3 gets throttled - level3b is decayed, since it has no more load - level2 get throttled - worker3 get unthrottled - level2 get unthrottled - worker3 is added to list - level3b is not added to list, since nr_running==0 and is decayed The attached diff (based on https://lore.kernel.org/lkml/20210518125202.78658-3-odin@uged.al/) fixes the issue for me. Not the most elegant solution, but the simplest one as of now, and to show what is wrong. Any thoughts Vincent? Thanks Odin diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index bfaa6e1f6067..aa32e9c29efd 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -376,7 +376,8 @@ static inline bool list_add_leaf_cfs_rq(struct cfs_rq *cfs_rq) return false; } -static inline void list_del_leaf_cfs_rq(struct cfs_rq *cfs_rq) +/* Returns 1 if cfs_rq was present in the list and removed */ +static inline bool list_del_leaf_cfs_rq(struct cfs_rq *cfs_rq) { if (cfs_rq->on_list) { struct rq *rq = rq_of(cfs_rq); @@ -393,7 +394,9 @@ static inline void list_del_leaf_cfs_rq(struct cfs_rq *cfs_rq) list_del_rcu(&cfs_rq->leaf_cfs_rq_list); cfs_rq->on_list = 0; + return 1; } + return 0; } static inline void assert_list_leaf_cfs_rq(struct rq *rq) @@ -3298,24 +3301,6 @@ static inline void cfs_rq_util_change(struct cfs_rq *cfs_rq, int flags) #ifdef CONFIG_SMP #ifdef CONFIG_FAIR_GROUP_SCHED - -static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq) -{ - if (cfs_rq->load.weight) - return false; - - if (cfs_rq->avg.load_sum) - return false; - - if (cfs_rq->avg.util_sum) - return false; - - if (cfs_rq->avg.runnable_sum) - return false; - - return true; -} - /** * update_tg_load_avg - update the tg's load avg * @cfs_rq: the cfs_rq whose avg changed @@ -4109,11 +4094,6 @@ static inline void update_misfit_status(struct task_struct *p, struct rq *rq) #else /* CONFIG_SMP */ -static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq) -{ - return true; -} - #define UPDATE_TG 0x0 #define SKIP_AGE_LOAD 0x0 #define DO_ATTACH 0x0 @@ -4771,10 +4751,11 @@ static int tg_unthrottle_up(struct task_group *tg, void *data) if (!cfs_rq->throttle_count) { cfs_rq->throttled_clock_task_time += rq_clock_task(rq) - cfs_rq->throttled_clock_task; - - /* Add cfs_rq with load or one or more already running entities to the list */ - if (!cfs_rq_is_decayed(cfs_rq) || cfs_rq->nr_running) + if (cfs_rq->insert_on_unthrottle) { list_add_leaf_cfs_rq(cfs_rq); + if (tg->parent) + tg->parent->cfs_rq[cpu_of(rq)]->insert_on_unthrottle = true; + } } return 0; @@ -4788,7 +4769,7 @@ static int tg_throttle_down(struct task_group *tg, void *data) /* group is entering throttled state, stop time */ if (!cfs_rq->throttle_count) { cfs_rq->throttled_clock_task = rq_clock_task(rq); - list_del_leaf_cfs_rq(cfs_rq); + cfs_rq->insert_on_unthrottle = list_del_leaf_cfs_rq(cfs_rq); } cfs_rq->throttle_count++; @@ -8019,6 +8000,23 @@ static bool __update_blocked_others(struct rq *rq, bool *done) #ifdef CONFIG_FAIR_GROUP_SCHED +static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq) +{ + if (cfs_rq->load.weight) + return false; + + if (cfs_rq->avg.load_sum) + return false; + + if (cfs_rq->avg.util_sum) + return false; + + if (cfs_rq->avg.runnable_sum) + return false; + + return true; +} + static bool __update_blocked_fair(struct rq *rq, bool *done) { struct cfs_rq *cfs_rq, *pos; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index a189bec13729..12a707d99ee6 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -602,6 +602,7 @@ struct cfs_rq { u64 throttled_clock_task_time; int throttled; int throttle_count; + int insert_on_unthrottle; struct list_head throttled_list; #endif /* CONFIG_CFS_BANDWIDTH */ #endif /* CONFIG_FAIR_GROUP_SCHED */ ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests 2021-06-21 12:42 ` Odin Ugedal @ 2021-06-21 16:22 ` Vincent Guittot 2021-06-21 16:45 ` Odin Ugedal 2021-06-21 17:09 ` Vincent Guittot 0 siblings, 2 replies; 13+ messages in thread From: Vincent Guittot @ 2021-06-21 16:22 UTC (permalink / raw) To: Odin Ugedal; +Cc: Sachin Sant, Peter Zijlstra, linuxppc-dev, open list Le lundi 21 juin 2021 à 14:42:23 (+0200), Odin Ugedal a écrit : > Hi, > > Did some more research, and it looks like this is what happens: > > $ tree /sys/fs/cgroup/ltp/ -d --charset=ascii > /sys/fs/cgroup/ltp/ > |-- drain > `-- test-6851 > `-- level2 > |-- level3a > | |-- worker1 > | `-- worker2 > `-- level3b > `-- worker3 > > Timeline (ish): > - worker3 gets throttled > - level3b is decayed, since it has no more load > - level2 get throttled > - worker3 get unthrottled > - level2 get unthrottled > - worker3 is added to list > - level3b is not added to list, since nr_running==0 and is decayed > > > The attached diff (based on > https://lore.kernel.org/lkml/20210518125202.78658-3-odin@uged.al/) > fixes the issue for me. Not the most elegant solution, but the > simplest one as of now, and to show what is wrong. > > Any thoughts Vincent? I would prefer that we use the reason of adding the cfs in the list instead. Something like the below should also fixed the problem. It is based on a proposal I made to Rik sometimes ago when he tried to flatten the rq: https://lore.kernel.org/lkml/20190906191237.27006-6-riel@surriel.com/ This will ensure that a cfs is added in the list whenever one of its child is still in the list. --- kernel/sched/fair.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index ea7de54cb022..e751061a9449 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -3272,6 +3272,31 @@ static inline void cfs_rq_util_change(struct cfs_rq *cfs_rq, int flags) #ifdef CONFIG_SMP #ifdef CONFIG_FAIR_GROUP_SCHED +/* + * Because list_add_leaf_cfs_rq always places a child cfs_rq on the list + * immediately before a parent cfs_rq, and cfs_rqs are removed from the list + * bottom-up, we only have to test whether the cfs_rq before us on the list + * is our child. + * If cfs_rq is not on the list, test wether a child needs its to be added to + * connect a branch to the tree * (see list_add_leaf_cfs_rq() for details). + */ +static inline bool child_cfs_rq_on_list(struct cfs_rq *cfs_rq) +{ + struct cfs_rq *prev_cfs_rq; + struct list_head *prev; + + if (cfs_rq->on_list) { + prev = cfs_rq->leaf_cfs_rq_list.prev; + } else { + struct rq *rq = rq_of(cfs_rq); + + prev = rq->tmp_alone_branch; + } + + prev_cfs_rq = container_of(prev, struct cfs_rq, leaf_cfs_rq_list); + + return (prev_cfs_rq->tg->parent == cfs_rq->tg); +} static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq) { @@ -3287,6 +3312,9 @@ static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq) if (cfs_rq->avg.runnable_sum) return false; + if (child_cfs_rq_on_list(cfs_rq)) + return false; + return true; } -- 2.17.1 > > Thanks > Odin > > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c > index bfaa6e1f6067..aa32e9c29efd 100644 > --- a/kernel/sched/fair.c > +++ b/kernel/sched/fair.c > @@ -376,7 +376,8 @@ static inline bool list_add_leaf_cfs_rq(struct > cfs_rq *cfs_rq) > return false; > } > > -static inline void list_del_leaf_cfs_rq(struct cfs_rq *cfs_rq) > +/* Returns 1 if cfs_rq was present in the list and removed */ > +static inline bool list_del_leaf_cfs_rq(struct cfs_rq *cfs_rq) > { > if (cfs_rq->on_list) { > struct rq *rq = rq_of(cfs_rq); > @@ -393,7 +394,9 @@ static inline void list_del_leaf_cfs_rq(struct > cfs_rq *cfs_rq) > > list_del_rcu(&cfs_rq->leaf_cfs_rq_list); > cfs_rq->on_list = 0; > + return 1; > } > + return 0; > } > > static inline void assert_list_leaf_cfs_rq(struct rq *rq) > @@ -3298,24 +3301,6 @@ static inline void cfs_rq_util_change(struct > cfs_rq *cfs_rq, int flags) > > #ifdef CONFIG_SMP > #ifdef CONFIG_FAIR_GROUP_SCHED > - > -static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq) > -{ > - if (cfs_rq->load.weight) > - return false; > - > - if (cfs_rq->avg.load_sum) > - return false; > - > - if (cfs_rq->avg.util_sum) > - return false; > - > - if (cfs_rq->avg.runnable_sum) > - return false; > - > - return true; > -} > - > /** > * update_tg_load_avg - update the tg's load avg > * @cfs_rq: the cfs_rq whose avg changed > @@ -4109,11 +4094,6 @@ static inline void update_misfit_status(struct > task_struct *p, struct rq *rq) > > #else /* CONFIG_SMP */ > > -static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq) > -{ > - return true; > -} > - > #define UPDATE_TG 0x0 > #define SKIP_AGE_LOAD 0x0 > #define DO_ATTACH 0x0 > @@ -4771,10 +4751,11 @@ static int tg_unthrottle_up(struct task_group > *tg, void *data) > if (!cfs_rq->throttle_count) { > cfs_rq->throttled_clock_task_time += rq_clock_task(rq) - > cfs_rq->throttled_clock_task; > - > - /* Add cfs_rq with load or one or more already running > entities to the list */ > - if (!cfs_rq_is_decayed(cfs_rq) || cfs_rq->nr_running) > + if (cfs_rq->insert_on_unthrottle) { > list_add_leaf_cfs_rq(cfs_rq); > + if (tg->parent) > + > tg->parent->cfs_rq[cpu_of(rq)]->insert_on_unthrottle = true; > + } > } > > return 0; > @@ -4788,7 +4769,7 @@ static int tg_throttle_down(struct task_group > *tg, void *data) > /* group is entering throttled state, stop time */ > if (!cfs_rq->throttle_count) { > cfs_rq->throttled_clock_task = rq_clock_task(rq); > - list_del_leaf_cfs_rq(cfs_rq); > + cfs_rq->insert_on_unthrottle = list_del_leaf_cfs_rq(cfs_rq); > } > cfs_rq->throttle_count++; > > @@ -8019,6 +8000,23 @@ static bool __update_blocked_others(struct rq > *rq, bool *done) > > #ifdef CONFIG_FAIR_GROUP_SCHED > > +static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq) > +{ > + if (cfs_rq->load.weight) > + return false; > + > + if (cfs_rq->avg.load_sum) > + return false; > + > + if (cfs_rq->avg.util_sum) > + return false; > + > + if (cfs_rq->avg.runnable_sum) > + return false; > + > + return true; > +} > + > static bool __update_blocked_fair(struct rq *rq, bool *done) > { > struct cfs_rq *cfs_rq, *pos; > diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h > index a189bec13729..12a707d99ee6 100644 > --- a/kernel/sched/sched.h > +++ b/kernel/sched/sched.h > @@ -602,6 +602,7 @@ struct cfs_rq { > u64 throttled_clock_task_time; > int throttled; > int throttle_count; > + int insert_on_unthrottle; > struct list_head throttled_list; > #endif /* CONFIG_CFS_BANDWIDTH */ > #endif /* CONFIG_FAIR_GROUP_SCHED */ ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests 2021-06-21 16:22 ` Vincent Guittot @ 2021-06-21 16:45 ` Odin Ugedal 2021-06-21 17:07 ` Vincent Guittot 2021-06-21 17:09 ` Vincent Guittot 1 sibling, 1 reply; 13+ messages in thread From: Odin Ugedal @ 2021-06-21 16:45 UTC (permalink / raw) To: Vincent Guittot Cc: Sachin Sant, linuxppc-dev, Peter Zijlstra, Odin Ugedal, open list man. 21. jun. 2021 kl. 18:22 skrev Vincent Guittot <vincent.guittot@linaro.org>: > I would prefer that we use the reason of adding the cfs in the list instead. > > Something like the below should also fixed the problem. It is based on a > proposal I made to Rik sometimes ago when he tried to flatten the rq: > https://lore.kernel.org/lkml/20190906191237.27006-6-riel@surriel.com/ > > This will ensure that a cfs is added in the list whenever one of its child > is still in the list. Oh, yeah, that is a much more elegant solution! It fixes the issue as well! Feel free to add this when/if you submit it as a patch: Acked-by: Odin Ugedal <odin@uged.al> Odin ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests 2021-06-21 16:45 ` Odin Ugedal @ 2021-06-21 17:07 ` Vincent Guittot 0 siblings, 0 replies; 13+ messages in thread From: Vincent Guittot @ 2021-06-21 17:07 UTC (permalink / raw) To: Odin Ugedal; +Cc: Sachin Sant, Peter Zijlstra, linuxppc-dev, open list On Mon, 21 Jun 2021 at 18:45, Odin Ugedal <odin@uged.al> wrote: > > man. 21. jun. 2021 kl. 18:22 skrev Vincent Guittot <vincent.guittot@linaro.org>: > > I would prefer that we use the reason of adding the cfs in the list instead. > > > > Something like the below should also fixed the problem. It is based on a > > proposal I made to Rik sometimes ago when he tried to flatten the rq: > > https://lore.kernel.org/lkml/20190906191237.27006-6-riel@surriel.com/ > > > > This will ensure that a cfs is added in the list whenever one of its child > > is still in the list. > > Oh, yeah, that is a much more elegant solution! It fixes the issue as well! > > Feel free to add this when/if you submit it as a patch: > Acked-by: Odin Ugedal <odin@uged.al> Thanks > > Odin ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests 2021-06-21 16:22 ` Vincent Guittot 2021-06-21 16:45 ` Odin Ugedal @ 2021-06-21 17:09 ` Vincent Guittot 2021-06-21 17:31 ` Sachin Sant 1 sibling, 1 reply; 13+ messages in thread From: Vincent Guittot @ 2021-06-21 17:09 UTC (permalink / raw) To: Sachin Sant; +Cc: Peter Zijlstra, Odin Ugedal, linuxppc-dev, open list Hi Sacha On Mon, 21 Jun 2021 at 18:22, Vincent Guittot <vincent.guittot@linaro.org> wrote: > > Le lundi 21 juin 2021 à 14:42:23 (+0200), Odin Ugedal a écrit : > > Hi, > > > > Did some more research, and it looks like this is what happens: > > > > $ tree /sys/fs/cgroup/ltp/ -d --charset=ascii > > /sys/fs/cgroup/ltp/ > > |-- drain > > `-- test-6851 > > `-- level2 > > |-- level3a > > | |-- worker1 > > | `-- worker2 > > `-- level3b > > `-- worker3 > > > > Timeline (ish): > > - worker3 gets throttled > > - level3b is decayed, since it has no more load > > - level2 get throttled > > - worker3 get unthrottled > > - level2 get unthrottled > > - worker3 is added to list > > - level3b is not added to list, since nr_running==0 and is decayed > > > > > > The attached diff (based on > > https://lore.kernel.org/lkml/20210518125202.78658-3-odin@uged.al/) > > fixes the issue for me. Not the most elegant solution, but the > > simplest one as of now, and to show what is wrong. > > > > Any thoughts Vincent? > > > I would prefer that we use the reason of adding the cfs in the list instead. > > Something like the below should also fixed the problem. It is based on a > proposal I made to Rik sometimes ago when he tried to flatten the rq: > https://lore.kernel.org/lkml/20190906191237.27006-6-riel@surriel.com/ > > This will ensure that a cfs is added in the list whenever one of its child > is still in the list. Could you confirm that this patch fixes the problem for you too ? > > --- > kernel/sched/fair.c | 28 ++++++++++++++++++++++++++++ > 1 file changed, 28 insertions(+) > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c > index ea7de54cb022..e751061a9449 100644 > --- a/kernel/sched/fair.c > +++ b/kernel/sched/fair.c > @@ -3272,6 +3272,31 @@ static inline void cfs_rq_util_change(struct cfs_rq *cfs_rq, int flags) > > #ifdef CONFIG_SMP > #ifdef CONFIG_FAIR_GROUP_SCHED > +/* > + * Because list_add_leaf_cfs_rq always places a child cfs_rq on the list > + * immediately before a parent cfs_rq, and cfs_rqs are removed from the list > + * bottom-up, we only have to test whether the cfs_rq before us on the list > + * is our child. > + * If cfs_rq is not on the list, test wether a child needs its to be added to > + * connect a branch to the tree * (see list_add_leaf_cfs_rq() for details). > + */ > +static inline bool child_cfs_rq_on_list(struct cfs_rq *cfs_rq) > +{ > + struct cfs_rq *prev_cfs_rq; > + struct list_head *prev; > + > + if (cfs_rq->on_list) { > + prev = cfs_rq->leaf_cfs_rq_list.prev; > + } else { > + struct rq *rq = rq_of(cfs_rq); > + > + prev = rq->tmp_alone_branch; > + } > + > + prev_cfs_rq = container_of(prev, struct cfs_rq, leaf_cfs_rq_list); > + > + return (prev_cfs_rq->tg->parent == cfs_rq->tg); > +} > > static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq) > { > @@ -3287,6 +3312,9 @@ static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq) > if (cfs_rq->avg.runnable_sum) > return false; > > + if (child_cfs_rq_on_list(cfs_rq)) > + return false; > + > return true; > } > > -- > 2.17.1 > > > > > > > Thanks > > Odin > > > > > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c > > index bfaa6e1f6067..aa32e9c29efd 100644 > > --- a/kernel/sched/fair.c > > +++ b/kernel/sched/fair.c > > @@ -376,7 +376,8 @@ static inline bool list_add_leaf_cfs_rq(struct > > cfs_rq *cfs_rq) > > return false; > > } > > > > -static inline void list_del_leaf_cfs_rq(struct cfs_rq *cfs_rq) > > +/* Returns 1 if cfs_rq was present in the list and removed */ > > +static inline bool list_del_leaf_cfs_rq(struct cfs_rq *cfs_rq) > > { > > if (cfs_rq->on_list) { > > struct rq *rq = rq_of(cfs_rq); > > @@ -393,7 +394,9 @@ static inline void list_del_leaf_cfs_rq(struct > > cfs_rq *cfs_rq) > > > > list_del_rcu(&cfs_rq->leaf_cfs_rq_list); > > cfs_rq->on_list = 0; > > + return 1; > > } > > + return 0; > > } > > > > static inline void assert_list_leaf_cfs_rq(struct rq *rq) > > @@ -3298,24 +3301,6 @@ static inline void cfs_rq_util_change(struct > > cfs_rq *cfs_rq, int flags) > > > > #ifdef CONFIG_SMP > > #ifdef CONFIG_FAIR_GROUP_SCHED > > - > > -static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq) > > -{ > > - if (cfs_rq->load.weight) > > - return false; > > - > > - if (cfs_rq->avg.load_sum) > > - return false; > > - > > - if (cfs_rq->avg.util_sum) > > - return false; > > - > > - if (cfs_rq->avg.runnable_sum) > > - return false; > > - > > - return true; > > -} > > - > > /** > > * update_tg_load_avg - update the tg's load avg > > * @cfs_rq: the cfs_rq whose avg changed > > @@ -4109,11 +4094,6 @@ static inline void update_misfit_status(struct > > task_struct *p, struct rq *rq) > > > > #else /* CONFIG_SMP */ > > > > -static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq) > > -{ > > - return true; > > -} > > - > > #define UPDATE_TG 0x0 > > #define SKIP_AGE_LOAD 0x0 > > #define DO_ATTACH 0x0 > > @@ -4771,10 +4751,11 @@ static int tg_unthrottle_up(struct task_group > > *tg, void *data) > > if (!cfs_rq->throttle_count) { > > cfs_rq->throttled_clock_task_time += rq_clock_task(rq) - > > cfs_rq->throttled_clock_task; > > - > > - /* Add cfs_rq with load or one or more already running > > entities to the list */ > > - if (!cfs_rq_is_decayed(cfs_rq) || cfs_rq->nr_running) > > + if (cfs_rq->insert_on_unthrottle) { > > list_add_leaf_cfs_rq(cfs_rq); > > + if (tg->parent) > > + > > tg->parent->cfs_rq[cpu_of(rq)]->insert_on_unthrottle = true; > > + } > > } > > > > return 0; > > @@ -4788,7 +4769,7 @@ static int tg_throttle_down(struct task_group > > *tg, void *data) > > /* group is entering throttled state, stop time */ > > if (!cfs_rq->throttle_count) { > > cfs_rq->throttled_clock_task = rq_clock_task(rq); > > - list_del_leaf_cfs_rq(cfs_rq); > > + cfs_rq->insert_on_unthrottle = list_del_leaf_cfs_rq(cfs_rq); > > } > > cfs_rq->throttle_count++; > > > > @@ -8019,6 +8000,23 @@ static bool __update_blocked_others(struct rq > > *rq, bool *done) > > > > #ifdef CONFIG_FAIR_GROUP_SCHED > > > > +static inline bool cfs_rq_is_decayed(struct cfs_rq *cfs_rq) > > +{ > > + if (cfs_rq->load.weight) > > + return false; > > + > > + if (cfs_rq->avg.load_sum) > > + return false; > > + > > + if (cfs_rq->avg.util_sum) > > + return false; > > + > > + if (cfs_rq->avg.runnable_sum) > > + return false; > > + > > + return true; > > +} > > + > > static bool __update_blocked_fair(struct rq *rq, bool *done) > > { > > struct cfs_rq *cfs_rq, *pos; > > diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h > > index a189bec13729..12a707d99ee6 100644 > > --- a/kernel/sched/sched.h > > +++ b/kernel/sched/sched.h > > @@ -602,6 +602,7 @@ struct cfs_rq { > > u64 throttled_clock_task_time; > > int throttled; > > int throttle_count; > > + int insert_on_unthrottle; > > struct list_head throttled_list; > > #endif /* CONFIG_CFS_BANDWIDTH */ > > #endif /* CONFIG_FAIR_GROUP_SCHED */ ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests 2021-06-21 17:09 ` Vincent Guittot @ 2021-06-21 17:31 ` Sachin Sant 2021-06-21 17:44 ` Vincent Guittot 0 siblings, 1 reply; 13+ messages in thread From: Sachin Sant @ 2021-06-21 17:31 UTC (permalink / raw) To: Vincent Guittot; +Cc: Peter Zijlstra, linuxppc-dev, Odin Ugedal, open list >>> Any thoughts Vincent? >> >> >> I would prefer that we use the reason of adding the cfs in the list instead. >> >> Something like the below should also fixed the problem. It is based on a >> proposal I made to Rik sometimes ago when he tried to flatten the rq: >> https://lore.kernel.org/lkml/20190906191237.27006-6-riel@surriel.com/ >> >> This will ensure that a cfs is added in the list whenever one of its child >> is still in the list. > > Could you confirm that this patch fixes the problem for you too ? > Thanks for the fix. The patch fixes the reported problem. The test ran to completion without any failure. Reported-by: Sachin Sant <sachinp@linux.vnet.ibm.com> Tested-by: Sachin Sant <sachinp@linux.vnet.ibm.com> -Sachin ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests 2021-06-21 17:31 ` Sachin Sant @ 2021-06-21 17:44 ` Vincent Guittot 0 siblings, 0 replies; 13+ messages in thread From: Vincent Guittot @ 2021-06-21 17:44 UTC (permalink / raw) To: Sachin Sant; +Cc: Peter Zijlstra, linuxppc-dev, Odin Ugedal, open list On Mon, 21 Jun 2021 at 19:32, Sachin Sant <sachinp@linux.vnet.ibm.com> wrote: > > >>> Any thoughts Vincent? > >> > >> > >> I would prefer that we use the reason of adding the cfs in the list instead. > >> > >> Something like the below should also fixed the problem. It is based on a > >> proposal I made to Rik sometimes ago when he tried to flatten the rq: > >> https://lore.kernel.org/lkml/20190906191237.27006-6-riel@surriel.com/ > >> > >> This will ensure that a cfs is added in the list whenever one of its child > >> is still in the list. > > > > Could you confirm that this patch fixes the problem for you too ? > > > Thanks for the fix. > > The patch fixes the reported problem. The test ran to completion without > any failure. > > Reported-by: Sachin Sant <sachinp@linux.vnet.ibm.com> > Tested-by: Sachin Sant <sachinp@linux.vnet.ibm.com> Thanks > > -Sachin > ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2021-06-21 17:44 UTC | newest] Thread overview: 13+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2021-06-21 6:32 [powerpc][5.13.0-rc7] Kernel warning (kernel/sched/fair.c:401) while running LTP tests Sachin Sant 2021-06-21 9:39 ` Odin Ugedal 2021-06-21 9:50 ` Vincent Guittot 2021-06-21 9:54 ` Odin Ugedal 2021-06-21 10:57 ` Sachin Sant 2021-06-21 11:04 ` Odin Ugedal 2021-06-21 12:42 ` Odin Ugedal 2021-06-21 16:22 ` Vincent Guittot 2021-06-21 16:45 ` Odin Ugedal 2021-06-21 17:07 ` Vincent Guittot 2021-06-21 17:09 ` Vincent Guittot 2021-06-21 17:31 ` Sachin Sant 2021-06-21 17:44 ` Vincent Guittot
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).