From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1B441C6FD1C for ; Wed, 22 Mar 2023 09:38:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Content-ID:Message-ID:Date :Subject:CC:To:From:Reply-To:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=5CXi6sJkZUvIcfJ0A8qOKKNmX+1hWll+hPtk80984ZA=; b=L/pPTnQEQowz0y 15ID5xLoqCGnYIrZwFZCozFJ80v+1c6S1v8rTIfVfdY8LLSnQH7mFJxT1szngW8ZnMJLRYjdyakw6 7ALWhuAjg+WZNLuQmkq0H2ZNfHpIcM0qwwgbOvek4DVfLX8FkEtMRiXIU4Om/KbByNbAWfTqKEPtt V3NL+1uah6vlrYT87vbUrMBbOJAjyIxLeUjd5B/3SEtURlmlc6Iu3O7esyVP4vfX8QUq304wiSM2g VQpoCkhAL9s2n7mzq9lHzw6uSt4c6vu7djRYlrTdQmzHnXWkAROBuH4Y70mtVH0vfndnNbV9IV48k YErWs/Czc/SLtEgzDfkQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1peuuE-00FSjX-2I; Wed, 22 Mar 2023 09:37:18 +0000 Received: from mailgw02.mediatek.com ([216.200.240.185]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1peuuB-00FSgt-0g; Wed, 22 Mar 2023 09:37:17 +0000 X-UUID: 1eb6f768c89511ed8687db9d93187ff1-20230322 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mediatek.com; s=dk; h=MIME-Version:Content-Transfer-Encoding:Content-ID:Content-Type:Message-ID:Date:Subject:CC:To:From; bh=gkAyObfBWvpmO3MTih/t4Sj8wL6a2RURSnjIbGgTIbQ=; b=fgzXhnG3zZ0cUuYi+znjWLNjLQheko2W4mCuIu3wHXtgKRHm/tz3UZm934FN/ItSN5XtGXLrevkYUXAOt+uDsOVd8hbxA8HEjPQGbwA7dO9gwjtLlDOP5IJDcyOKTGSzLwO7uvAk5coRYb1rzLpY3vxTSzgk9KcCsk7XweDn7iU=; X-CID-P-RULE: Release_Ham X-CID-O-INFO: VERSION:1.1.22,REQID:d8c7e957-ed1d-4bb0-b947-a58498bd9c19,IP:0,U RL:0,TC:0,Content:0,EDM:-30,RT:1,SF:0,FILE:0,BULK:0,RULE:Release_Ham,ACTIO N:release,TS:-29 X-CID-META: VersionHash:120426c,CLOUDID:003805b4-beed-4dfc-bd9c-e1b22fa6ccc4,B ulkID:nil,BulkQuantity:0,Recheck:0,SF:102,TC:nil,Content:0,EDM:2,IP:nil,UR L:0,File:nil,Bulk:nil,QS:nil,BEC:nil,COL:0,OSI:0,OSA:0,AV:0 X-CID-BVR: 0 X-CID-BAS: 0,_,0,_ X-UUID: 1eb6f768c89511ed8687db9d93187ff1-20230322 Received: from mtkmbs11n2.mediatek.inc [(172.21.101.187)] by mailgw02.mediatek.com (envelope-from ) (musrelay.mediatek.com ESMTP with TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 256/256) with ESMTP id 1977684437; Wed, 22 Mar 2023 02:37:10 -0700 Received: from mtkmbs10n2.mediatek.inc (172.21.101.183) by mtkmbs13n2.mediatek.inc (172.21.101.108) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1118.25; Wed, 22 Mar 2023 17:37:08 +0800 Received: from APC01-SG2-obe.outbound.protection.outlook.com (172.21.101.239) by mtkmbs10n2.mediatek.inc (172.21.101.183) with Microsoft SMTP Server id 15.2.1118.25 via Frontend Transport; Wed, 22 Mar 2023 17:37:08 +0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IzfnAG8WFrAp78lYFNdtpxpDlTXMvEYXMt55YZP/KCFLgoYAvqRGV50iQH2H9GvCdFu75UXFbMxB4yVscAknDFuN2ayZs4ozQzbXZdhA7AFaxROFLAkABgUfVfUjMnAMA1sroJgNCLDTpnQ0ftaVtmIyAJEKaknT1jA2ZWru8oaMPUe3UQMzgSYlWUaoax4iJdWT+l0Ykxdc32FPKHua2CKvPPJ0bUxonHuECVcDk6bXNk4mZ8xT97lOp9IucQao1HwyjewZizGgGdkVVI2z0ox3FIp1A+A7RCdaD8UjrVHzYGcQ3llObN+9UBF133v/G/pc/5UyEjIeGhMeH/YR8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=gkAyObfBWvpmO3MTih/t4Sj8wL6a2RURSnjIbGgTIbQ=; b=giDgOHNyF5RgltpaUuUKp+45NzG4uKGGHPEkXBt33Nlb8XAmOuZ83jtdJ/LqsPh4LFGucAJIr2FLluJvG1759mlBnlc9xJBSUO15jJtJM97IYYjNgb98+G84K9pGyxMwsLZpmmM20p5XqnBlx+WG+bPvWsmD0ba7jS9ETS7r2jA5zXpyrgSXjJ67SIToz6LdzUomKJ92IvfME+D2pDbQQUPXtUtzIlkA4KjmNmtj8ORw0H/wU0taoAAKeD0ZsqPMaUj+0NJzzVs2yQCPpNVU67PwH0UhhZGMxPST443orzbG1GzJHC13ySTJRHgi3QGDSgClF9I66erCSCSfHl7gGw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mediatek.com; dmarc=pass action=none header.from=mediatek.com; dkim=pass header.d=mediatek.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mediateko365.onmicrosoft.com; s=selector2-mediateko365-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gkAyObfBWvpmO3MTih/t4Sj8wL6a2RURSnjIbGgTIbQ=; b=G7OOqRj89OIHv5vDSsfRRBY++ezY4RpzLFPhyBGqTWrdNTwHoaqL5gwBjuwheOPDYZVvPFfq59O8XG1l1V0UWryeUKzE50QB8WaKBpcc3yqmQXE6bVbPzKJkhMgrm3qiYDt5/eqIdJBYEGvtEQAA5D53chBOp4VQhJq5OIoTi7Q= Received: from SEYPR03MB6626.apcprd03.prod.outlook.com (2603:1096:101:83::7) by KL1PR03MB5836.apcprd03.prod.outlook.com (2603:1096:820:81::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6178.37; Wed, 22 Mar 2023 09:37:02 +0000 Received: from SEYPR03MB6626.apcprd03.prod.outlook.com ([fe80::f497:1db:7e78:fcfe]) by SEYPR03MB6626.apcprd03.prod.outlook.com ([fe80::f497:1db:7e78:fcfe%3]) with mapi id 15.20.6178.037; Wed, 22 Mar 2023 09:37:01 +0000 From: =?utf-8?B?UnlhbiBYaWFvICjogpbmsLTmnpcp?= To: "tj@kernel.org" , "peterz@infradead.org" , "vschneid@redhat.com" CC: "linux-arm-kernel@lists.infradead.org" , wsd_upstream , "linux-kernel@vger.kernel.org" , "linux-mediatek@lists.infradead.org" , =?utf-8?B?WW9uZ2p1biBMdW8gKOe9l+WLh+WGmyk=?= Subject: BUG: HANG_DETECT waiting for migration_cpu_stop() complete Thread-Topic: BUG: HANG_DETECT waiting for migration_cpu_stop() complete Thread-Index: AQHZXKHaCEisPEX/hka7ctaBtar8UQ== Date: Wed, 22 Mar 2023 09:37:01 +0000 Message-ID: Accept-Language: zh-HK, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=mediatek.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SEYPR03MB6626:EE_|KL1PR03MB5836:EE_ x-ms-office365-filtering-correlation-id: fb54c432-576e-4710-0f7a-08db2ab8fd61 x-ld-processed: a7687ede-7a6b-4ef6-bace-642f677fbe31,ExtAddr x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: HSjEIrzWtitCDgAHoDlzHgh3SRR9eYXJWf+7XX9Xoa9/b41RpEs7/2QNyEYR6dbRtjYDtrpEeBShXCcVcppr29sPJdj3oHHFK7SCHnYS4CoHOvaQCVbjYjnDMJZp3q320m740TEwu2LxFDYXRZjuxZpPp12Do/fOMjSFrnMLmvqsB0sEpHRcVBH8dZA3Sb/e2fVoPlTy9+9gOY/hVJotp91Sa7iliareI9dZJcKmevcOLDIcCWJ5pCp6oosHkZw9Mk4jsmAxO8gPvNu0bsE4pVWuNJQXscfkhu3XrywbSLNdn2SI0mWGhrA3fGWEG4FXG+hH2bZKRfAlIb1AnhyfwZ1Q+9IYz0le9tFCzl92dxZ4C0g667dTPrMrRf7l/ejzIAjQxXzNYf2wbZjoaBxvq1C0e3Hes/24diJN9/TNq3kkEDDB998N1aNVb1lQYJSVUmB5IZ+6EKDWSIK8bIUKAthPO9vbJB8FhwkiypEfJ7Ng/slUxQxkd59i41PVLCpVKEHQUy4lM4CgxenXqybb6tZTYw7nujrzd0Z1IR9BmGqiV2sHPI5qTbuHPPBo9D1S5wGiGvjLYRBVb/M18YiE9LVWSKkwKtCSoBhMeK3WE1xR3f1isx+3OSvRruAXXndd5gXWSxMfye8LzpnRZp2TNpbwHzlQPGK28+FrwE1HDsBMhWH2S1NyTHseEuRBoonilKaHje3DmiHjyyh5MgjVuwJ82UyeeAedIAO9VQUo1jU+K94P4jemSKW9qOMRdB6K x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SEYPR03MB6626.apcprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(346002)(366004)(136003)(396003)(376002)(39860400002)(451199018)(2616005)(6512007)(54906003)(86362001)(91956017)(38100700002)(38070700005)(122000001)(110136005)(71200400001)(66476007)(8676002)(64756008)(85182001)(36756003)(66446008)(2906002)(8936002)(4326008)(66556008)(66946007)(41300700001)(76116006)(5660300002)(107886003)(6486002)(6506007)(26005)(316002)(478600001)(186003)(83380400001);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?utf-8?B?THZjRnc0WHJGcVBUYzBMWXd6WEY4MDE2R2Y5cUNlSk56ajJBc2p2Mmc3a0hj?= =?utf-8?B?OXRlZmYvTXVJWURqcEJmVkR1SGw5cVhEdUZKbE1UV0VFTVNxcnhvbG54elBB?= =?utf-8?B?ZXN4ZkVuZ1hETzZyZzFTaE52cnpJTVhaY3R3eXpXSGZDZG1XbWNuUjNSbnlx?= =?utf-8?B?dVNTNFNRbldKSUdGZGxJaTVKYVlVZFdrWnNzZ3NRaVZjNjJVa3RmaUZER2E2?= =?utf-8?B?aVpkdEg4cXdHYlo2dWQ0UmJNK2J2bnVQTzcreTI1TnQ0OWtJcHkyV25iendV?= =?utf-8?B?SDZ4bzJENklGYTJRZWRvN2VXeVlFUzRRVDRLNlJHTGxZS0QveEdiSmEzcGFu?= =?utf-8?B?c2FyYXNVbHFCR09YSmJ6aHpSVW9lWjFjTmw4VHFHSEJWM3lnbkJ3Mnp0YzlV?= =?utf-8?B?eTFUUkwva1dPS2J5WVlVbWhPWHFOaHBNQ0VxUXRSQUppNUVnbVZub213N1Yr?= =?utf-8?B?M2E0TndWV20vamowcUFET0FYd054Vm12MnhUMkhuZjVkSEZnaFNrU2V0eks5?= =?utf-8?B?UDF6Wk94WUhYNjN2NEJHWTZvYzFXRTJBcUpLNlhzd3JBR0dCazRFbEVOc3Y4?= =?utf-8?B?NWt1UDR4YnlWa2prSzZMR3hhYlYvWFdTSTRKYVg1TTQwdFNZQ204aVZGUkti?= =?utf-8?B?eVlBNExpKzd5OUNsUTNzQ0FGaUhyYjY1alhReFdkb1pvU2gwSElOT1p1Vnp4?= =?utf-8?B?TUFBVHF3QkUwSWZ2Y3Zta3hMN0F6cmtuZURleXhGRFdhdnFKSDlaZHNSTndk?= =?utf-8?B?eWJ3VHFXVDdGOVNOSWhsWGFWZVNkZ21JK2k3UVk5d0JBN3FNclNPL1R4V2Fw?= =?utf-8?B?NkFyVCswNjF3MVQ4cFdiVUdsQWRZVzRsMFNrY1phRVpxK0kwcElGNzRpdHRC?= =?utf-8?B?bHFKT1Boa1hqVVZkK2tmbkZKWTlHT29nazF2ZHhaVGZTN3lTOVJqekxBOExN?= =?utf-8?B?blAwaXEzenpscS9vN1IrZzJkSUkxbi9xL2o2a25PbmhwWkc0VkV3eExVeTM2?= =?utf-8?B?VUxZbkd6Q2tyRVFrTzdwUGlGSC9pUEowbUtpQXVrelRVSXVVWEdFMGEyc3FH?= =?utf-8?B?ZjNZV2x0VmRDU29FdDY2RmJtNnZvNjNUQkZKN20wbW1CWGc2Q1VpeWxzL1c1?= =?utf-8?B?N1BqSlJoQk8vdXJ3YW9FS3VabUY0OWFtV0lLcW9jbkh6RktkT09uL3Q4QWFZ?= =?utf-8?B?MUNhNjBoS05XeTcvQTlIeUVGMlJYZmc0TDMrYU9PNGFOT2tORmw1c0UvYWZa?= =?utf-8?B?cFJBKzA4cFFzSG13VWwzSkR1OHVQU3dKbXQzN0FRby9SRUFYNEdmdlI0eUtY?= =?utf-8?B?N2VINzRXTWpOTUR5VWFMZTczL3ZQODJtZ3dzR2JjaXVqSWZCMElwanh5VkZj?= =?utf-8?B?cXdtOXBKS1phOFh4S2htMWlJei9PTlVEaTUwOGVjMGp2N21hT3BaTFcvVjNx?= =?utf-8?B?RUJ6aEt0NmFObXBoNU11ZFJObHlCWmY4eXlqMnF0elJXNlBnMzRZTkF2VmtD?= =?utf-8?B?SUh3V1kvZzRnL3A2a1RrNUVSK0s1UEJrOUUvcFZLV3pDNmNsaElub280emRV?= =?utf-8?B?SzFUYzE2cmtOekhjQnNKUjZTSVFHT3p0MWJ0MlBYYUhxRTFtdVBzb1Ywa1NI?= =?utf-8?B?ZUt4ZkhySmpSaENxQlZ6N21OZW9NRmJ4aURIQmp5Y2IvUFI0TnJNMnBQWjlQ?= =?utf-8?B?aWdheURGWUpucWh0dkNiWDYwTjVjZTNXYmxOQjU2RlFPYm16YUk4QmlNSFlO?= =?utf-8?B?cFVGbkplVmJCS3ExTFc2a1BDL1dhTmZRNTNaeVFiUXJEWFd3V3JwckRKelFY?= =?utf-8?B?dFllTjRoM012Ym5VMjFab0lueDF3MDhGMlZUNWdEeUF4Yy91Q1Z1Z283S1h4?= =?utf-8?B?YzRxTTFrT2YxYUxXNEpSTjd1WmlmUEVZNzAzZkx0L0FzLzljMDdDLzI0Qklm?= =?utf-8?B?N3g5ZGt6c1VqcnY0NWQvdDdTNmRtekhDNlJFcU8vL3R2a00xckp5YTVYZThG?= =?utf-8?B?M24zS3l5cHpNWHJFSGxML3VnRTVENWVDUGo1blJISEhGRmV2VWtrS1J2TVBv?= =?utf-8?B?VmtvNCsvU0NUY3VTclVYZkxYcU9xbUNtbUNqbUdlRFNxZ04vS3JsaHpkSS8r?= =?utf-8?B?YkFQamkwa0w4QjcwU0paY3VScng3Sks1ODZZUysvYXNJRFVZS0JZd2lHNDhC?= =?utf-8?B?Z3c9PQ==?= Content-ID: <6CB9A53FEDC0C445BCADAADD7142B0ED@apcprd03.prod.outlook.com> MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SEYPR03MB6626.apcprd03.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: fb54c432-576e-4710-0f7a-08db2ab8fd61 X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Mar 2023 09:37:01.5934 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a7687ede-7a6b-4ef6-bace-642f677fbe31 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: aDXRswiuEkU/kXjA8zhJSdYRZspud//gvaDv5lVXcqkebQg1BPiN1SMrIcB2GeR5a9ZTsBf1+FCk4HDWQkDBxQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: KL1PR03MB5836 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230322_023715_597633_561ECE76 X-CRM114-Status: UNSURE ( 9.18 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi, We meet the HANG_DETECT happened in T SW version with kernel-5.15. Many tasks have been blocked for a long time. Root cause: migration_cpu_stop() is not complete due to is_migration_disabled(p) is true, complete is false and complete_all() never get executed. It let other task wait the rwsem. Detail: system_server waiting for cgroup_threadgroup_rwsem. OomAdjuster is holding the cgroup_threadgroup_rwsem and waiting for cpuset_rwsem. cpuset_hotplug_workfn is holding the cpuset_rwsem and waiting for affine_move_task() complete. affine_move_task() waiting for migration_cpu_stop() complete. The backtrace of system_server: __switch_to __schedule schedule percpu_rwsem_wait __percpu_down_read cgroup_css_set_fork => wait for cgroup_threadgroup_rwsem cgroup_can_fork copy_process kernel_clone The backtrace of OomAdjuster: __switch_to __schedule schedule percpu_rwsem_wait percpu_down_write cpuset_can_attach => wait for cpuset_rwsem cgroup_migrate_execute cgroup_attach_task __cgroup1_procs_write => hold cgroup_threadgroup_rwsem cgroup1_procs_write cgroup_file_write kernfs_fop_write_iter vfs_write ksys_write The backtrace of cpuset_hotplug_workfn: __switch_to __schedule schedule schedule_timeout wait_for_common affine_move_task => wait for complete __set_cpus_allowed_ptr_locked update_tasks_cpumask cpuset_hotplug_update_tasks => hold cpuset_rwsem cpuset_hotplug_workfn process_one_work worker_thread kthread In affine_move_task() will call migration_cpu_stop() and wait for it complete. In normal case, if migration_cpu_stop() complete it will inform everyone that he is done. But there is an exception case that will not notify. If is_migration_disabled(p) is true and complete will always is false, then complete_all() never get executed. static int migration_cpu_stop(void *data) { ... bool complete = false; ... if (task_rq(p) == rq) { if (is_migration_disabled(p)) goto out; => is_migration_disabled(p) = true, so complete = false. ... } ... out: ... if (complete) => complete = false, so complete_all() never get executed. complete_all(&pending->done); return 0; } Review the code, we found that there are many places can change is_migration_disabled() value. (such as: __rt_spin_lock(), rt_read_lock(), rt_write_lock(), ...) Do you have any suggestion for this issue? Thank you. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel