From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id D4D9A1F6692
	for <linux-kernel@vger.kernel.org>; Thu, 24 Oct 2024 15:50:12 +0000 (UTC)
Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=205.220.165.32
ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116;
	t=1729785014; cv=fail; b=iX9KjoPpZscHOWnTwZfHEh6SGhOZkPPFMZ4HYTMLr0GFYpm+cEqr5YmSe7AxqmbNURU/QRyNocv7pylMLAMcB+3YKDSwtPGJVuCiHrRN3Pi2Q6MYYBl78r5VV217Z9kM5L9WvqAInubSnw2uarD+INROBW/r8Hiwiud6LEaPCKY=
ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org;
	s=arc-20240116; t=1729785014; c=relaxed/simple;
	bh=Lxev65DswyWtcaDXgsqTy7U/sHO151GKUojE7a9NgCc=;
	h=Message-ID:Date:To:Cc:From:Subject:Content-Type:MIME-Version; b=rX7jbEnsUTvK+EWJLE12zDzoXQvO8/BqQy/Z4I2Eq+CnClAG2Fc8vlk2iiAD2yLuIT94mFSZcpChfw7zMJdYLkLV7n/JZdo7SwGtwCITyp2VPTu4bE12ZduaeTOK0LGxC4gauHHzR91G9C3HTL8+RjWQmk99huoH/7bIByMGLVM=
ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b=XEse4gr9; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b=gmiAh8w2; arc=fail smtp.client-ip=205.220.165.32
Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com
Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com
Authentication-Results: smtp.subspace.kernel.org;
	dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="XEse4gr9";
	dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b="gmiAh8w2"
Received: from pps.filterd (m0333521.ppops.net [127.0.0.1])
	by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 49OC1YCR030561;
	Thu, 24 Oct 2024 15:49:50 GMT
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc
	:content-transfer-encoding:content-type:date:from:message-id
	:mime-version:subject:to; s=corp-2023-11-20; bh=QVsV0Tn5vncxuwoD
	Q3s/3eCoAc6IVFSxbA69HkzFrqI=; b=XEse4gr9GI8FnY3jFT/WjKoW576+pmJU
	PytR2cqzR65SzNeRDwVXjM66Y2jiVb72s0kOwVIaV1OGvXpOHBLejgfV9akyuUQG
	oPVwuTP19nvzLUWUEnaJIZs0/vvXyFNsv3f/rufFKFLrpBs1GlcPC7Kr9F55/PIa
	QAMa5j+KiKRInPlm47nAqcAzbw4TOCVhLmwFS2+QZRrjWEyUa6T3vQfcl2l0qzqb
	ZSJMMp6uGlieoDDa8upx+XvGKxfesG+mKjDLIEekThhWzRti+gS7zGausf9fyyXw
	NQC6FCxHOn13sJKzWcN9IIiBVavaRJWihZQ7huf1me6WlPv4llEuFQ==
Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223])
	by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 42c53utyby-1
	(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);
	Thu, 24 Oct 2024 15:49:50 +0000 (GMT)
Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1])
	by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 49OFD10T027557;
	Thu, 24 Oct 2024 15:49:49 GMT
Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2172.outbound.protection.outlook.com [104.47.55.172])
	by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 42emh4h487-1
	(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);
	Thu, 24 Oct 2024 15:49:49 +0000
ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none;
 b=c8uHSl0+EYDnVXJiyOcI0ZSNAGiIXl8NtvsaPemGpPMCdYmzw0cIPHeuDpuImtOYMFKPcmpOgdru1by27RGwCg0R82z4ex1F0Gi88G2TKpZyYAXiKzM9gywGReFn2Xl3vuTOZZxfW0A24xl9eLzBVNfBVdJVArY301c6rYBWp7UbZJmB519rEy+SKbMUwTN7HoBpFbgRNQXfiFL5bEgzBV4i6pU63hlWLBJmTEl77pqbzqIzr9sGm5moRl+DzHIcBfeGv3E3JUhhYWkigMNxSktVvFt0sdghJiunHTKfDD2uo0a2vjSyK672J7JoMvh+JziWmu1UXAG9JyAybcVL1g==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector10001;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=QVsV0Tn5vncxuwoDQ3s/3eCoAc6IVFSxbA69HkzFrqI=;
 b=PGUz9JT0bnRVKFYIUZ/yO9VWj4dkyQfO75gsgf7+vcn+jaWMsSAiOk4ObhFKN2xnBlJ56jlY7gLEQQNk9+ygh7nR65zvkzobLHVdyUDDuAw0v5o9CfjyIqoeBuDnxge55zEhltdeMS/9SHgAmzTuRyScedHYx/UvTVmhAq0iJQqE35mwrQlcMAHrMbvU2/AYRVeQr6kVZu2mC/ngjaPwI6ajww0oOCXc5uKaBmcTRjqjnvsO1g5jILv/Y4Te4FKEQQibyoLkZthSRIXHIdyq9g58x706PPFxq9PrMPNm8vHnV9unbi+auCW/MPIRvKOBFXpyeD9h2J0JwNksesBFLw==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com;
 dkim=pass header.d=oracle.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=QVsV0Tn5vncxuwoDQ3s/3eCoAc6IVFSxbA69HkzFrqI=;
 b=gmiAh8w2IYVkZe21u1swE1ezPlRaSFQaOITcVF5Q8S9ncBkocvFZ09179sKFd8126WCG4ERSBGHOpfSqc1nsS2QRtvJLoPhqR54jM1IwPnYKfX14IhH/L8UDa06xjfVmOyZptvXIHEgynQ4+dljGqYQhjSJBSFJG9gD5tghfK5E=
Received: from DM6PR10MB4313.namprd10.prod.outlook.com (2603:10b6:5:212::20)
 by SA2PR10MB4729.namprd10.prod.outlook.com (2603:10b6:806:11c::9) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8093.21; Thu, 24 Oct
 2024 15:49:45 +0000
Received: from DM6PR10MB4313.namprd10.prod.outlook.com
 ([fe80::4f45:f4ab:121:e088]) by DM6PR10MB4313.namprd10.prod.outlook.com
 ([fe80::4f45:f4ab:121:e088%5]) with mapi id 15.20.8093.018; Thu, 24 Oct 2024
 15:49:44 +0000
Message-ID: <70cb224f-a468-40a0-82c7-735c590bdb43@oracle.com>
Date: Thu, 24 Oct 2024 16:49:40 +0100
User-Agent: Mozilla Thunderbird
Content-Language: en-US
To: tj@kernel.org, jiangshanlai@gmail.com, mingo@redhat.com,
        peterz@infradead.org, juri.lelli@redhat.com
Cc: jack@suse.cz, david@fromorbit.com, akpm@linux-foundation.org,
        linux-kernel@vger.kernel.org
From: John Garry <john.g.garry@oracle.com>
Subject: workqueue lockup debug
Organization: Oracle Corporation
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-ClientProxiedBy: LO4P265CA0149.GBRP265.PROD.OUTLOOK.COM
 (2603:10a6:600:2c7::13) To DM6PR10MB4313.namprd10.prod.outlook.com
 (2603:10b6:5:212::20)
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-TrafficTypeDiagnostic: DM6PR10MB4313:EE_|SA2PR10MB4729:EE_
X-MS-Office365-Filtering-Correlation-Id: 0578a776-52ab-4883-b493-08dcf4437ab3
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014;
X-Microsoft-Antispam-Message-Info:
	=?utf-8?B?YjN1dDZSalMwUmtOTm1ZTTVuOEdzbFNEWThscVBLcFloYnM5YUJudTF1eXl4?=
 =?utf-8?B?U2pGQktCMTI5UHFqcHdhZVVzY1JPYU92SGdSRElYeVN2R3JMRlY2Nkg1Mm54?=
 =?utf-8?B?Q21BR3RWL2tSd3VRZVVVV3BmUENhbWdSQ3I0M3EzaVo0bmlibi9FTWt2dWY0?=
 =?utf-8?B?bG1aUWFHQVRGRWZlMEJFRnNkM0ZqNUwvTEhYMlZzbDZGUDAyZXdib2oxZVlC?=
 =?utf-8?B?NEdrbGpqbVVKRGZJUDlOSUFHbC94UVNzNklHdDVnTFY3ZEl3R0dYMExkVWFO?=
 =?utf-8?B?SjkzUFlkcFZ1dmljOExJQUIwUUM5SGxSK0RSckkrSXVqY1pmM2h0ckpiK25l?=
 =?utf-8?B?S25tdERpUk9MMmNnUEtBdi92bEJHUkwvbUFIM29MTnZPMnVwOGNFdUZGaHQw?=
 =?utf-8?B?YWt5NkxBM1BlcUhHYXpKQStuSG1iZUJXVXRJTE14ZnRsbmFoWEpaSnhCS0JC?=
 =?utf-8?B?cHFxd1J0LzBPRnloNjJEZENXb2YwZmJ0aEV1SUdxcDZmczNaenVQaXZjR3Uw?=
 =?utf-8?B?alp2c2s0ZTdBczEya091R2hONmppUzYyaGFnM2J5SmwwVmJ0U0RDQUo2dFJq?=
 =?utf-8?B?QVloWmwvVXJUNk8vWTZSUFdHRng5VjEzajNIellndGdoS3YvY25za0NMUEVm?=
 =?utf-8?B?Qy90T3AwbENvczIxNU9RZmZScklOR0VIT3JXZ1lGSVpZZTlIOTBuRmxmbkgv?=
 =?utf-8?B?UXhhOCtXM1hnWit5OFRVVUkwaGRkZys3ajA1c0EvUlRwcEQxMWNic3g2SWdj?=
 =?utf-8?B?M3FjaDFmeWh6OUVmV3laMjRla1ZBejY0N0FxS3lhOWtwNWFBRlVvRTBpcHVI?=
 =?utf-8?B?SitiTkkvemZmK1MyaFE5ZmI4VE5QM0ZLK2JKeXBIZ1AwS2VEYThmK0FCN0JM?=
 =?utf-8?B?TDRRWFcxOGdrZy9oNjZXMnJYSFJtQU9IL2hMUEVuODVJZE9adit1WnF1WFVJ?=
 =?utf-8?B?RndPWVArZGtBWkF5eTk1dXBWcHdsZ1dvUEVVaU00L3RLRGdvVHRiazNqQm1Y?=
 =?utf-8?B?Z3JMUzRPam1oYUxMUlVZeEpVRTJ0Q3lUL0FBM2NHamdGZnJPblJLdlRMNUt6?=
 =?utf-8?B?KzgveGREZjY5c3NYdXhlQWhXRlV3SGxkdTNXSkxTV2wwQ0d6clorMndrU0dJ?=
 =?utf-8?B?d1NBclhMZ3I5VTA5WU82VFdpTXZIQ1NHMUZzZ2M2dVd6ZEprd0l3TmtlK2xK?=
 =?utf-8?B?MXhGZDB6MUpFTDgvN2RubnR2Ym05R3NnSlFEMTdqQ2ZISXlsT0hsOHl2S01j?=
 =?utf-8?B?bmpPYzBxTTR1c1R4VmxjRzNPL1J4NVBueUpNeGNQTDVWbi95NUczaENZWEpC?=
 =?utf-8?B?S2RnTmJ4d2haWUx3MTBuem1uMnhEaTNERGMwTmJmS0tFR1NUcC9zdmpKSGxa?=
 =?utf-8?B?d3A5bFJiNFVNRkFXa1pCN3ZhUUplR0IvRnVVMW1WMUxubVZ6cWVaeGNEOU41?=
 =?utf-8?B?L25VeWNySm5xRmpNekRrYVFsWU9aODNwbXJpU3REb25UZE1vSmZROTVxaVd2?=
 =?utf-8?B?bWFENnMwWFBRRzgyYVF3Zkl0S3lWZXp3YVgxSWFQQmdUNHBNZlAyR2h1d1kv?=
 =?utf-8?B?SURSWkMvMlIwOG5TZEJ1Wk90NTdoWTBoZC9NRXZKWExwMWJXWEkvVTJXSi9Y?=
 =?utf-8?B?Z1EzRzlLVjhTMFd6ZUMxc21sVEwwc1JpY3FFN3h5cDhtTFIxcDlKUUpBWE1P?=
 =?utf-8?B?VEVlSFFsWENNZVZSdkNxNFAvbHgxa3VqOW0wS3U5bGVKNUd0ZzJjWjIwUGJI?=
 =?utf-8?Q?im5nABDxy2r3uy/7PxL5S9U2ro9byLeR+Q19x6I?=
X-Forefront-Antispam-Report:
	CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4313.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(366016)(376014);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?Qmw3LzlYdmVqY3dpenlSVHo0QjNvVGRPbGNub0lpUEhvWk5ZRTZRVFVHZnk0?=
 =?utf-8?B?dllwV3Q2RlkxbFRxV2s5UHJDWHM3dGRCQ3NhZ2pPbUFvN3hTN2htb2hZaWIy?=
 =?utf-8?B?c0RmRVZINWs3TkRVMDJBN0xrbDB1UmxHNDN3RUNlQVlhZDhnWmF1ZGVLWC82?=
 =?utf-8?B?TWp0S3hjZG8zUHpjU2l5WmhzRVplMkdPWEJSMTVrcFFDc1R3Ry8rNDlLTXBL?=
 =?utf-8?B?NGhnVjFDbk1pUWtTYUVnNkM0Q1UvaUpBdGkvekF1UHhmamRmNk9mQ3VlcytH?=
 =?utf-8?B?alFIRFVHdWpqdzgwSHNPTU5hRGFNUHZwNnpiQXE5TmtpczJEUm1nVW5uYjIw?=
 =?utf-8?B?UFNlMW1NRm1uVUxlMzlxaWU1QWdvd3FETGI3eGgranhMNlFERTV5VWtmL3NH?=
 =?utf-8?B?WXNYWFFsZ21LZXdYNHluWGNMUDViQTdNT0tQQWVBdXJaQTRVa2NsMzVWVUZD?=
 =?utf-8?B?eFRsWWZJVzhhTXBFWW9CZVQ5M1hqc0dUdEZZLzVWM3NkVWIrcXZ5aWRQOGUy?=
 =?utf-8?B?b0VHWldPSkVheGhPZmpPNmpPdThSRlovM3VXTDE1YWZIb1F2MFd2OEh4Y2Jy?=
 =?utf-8?B?SFdkMHpCeG5tdi9mYWFQVG9xSWE3RjYvMmRGbllLUGtreXBWODBrTEhrbFJp?=
 =?utf-8?B?S1NZd3ZBTC9XTVZXR3JFb0dnWmhjMlpSUURYQUhFNTV1anlkemk4OGFDVGFM?=
 =?utf-8?B?dVRZL1hTMWUyVmNLV1U4dDkvd3hnYUI1eUl0ZEdtY2NBNi9zUkVVRU5zMmZR?=
 =?utf-8?B?cHMwbzlIWlpyc2dQS3ZMUHZXZk91R21yMWNhdThTRG50WHB1cDBETXd3bUNy?=
 =?utf-8?B?T3M0bXNsSnhSS3RoZHMzRnJLSWNVRWVpUjZLTmhON2tvTHlCblBmSEJTYnZR?=
 =?utf-8?B?S3ZBZk9BUk5wdUlzZEZ2TXlxT1BicncxRGxyTEFUeVV3UFFyYXQ0QjdYWkZP?=
 =?utf-8?B?U3AzblB2NXQzNyt5dVZGNXZKbUJudVAyZWlTYWlyQXJmNDdjU3h1V3JlQWxF?=
 =?utf-8?B?bjdXNjByeC92UCtpVHNCL05EWTZqblZHQlpIWFFpMUszYzhzdC85SUkvSE9v?=
 =?utf-8?B?WDYvcjdGdXk5NTJESnlBMENnb2hlbXhyYUZKSTVnRTNqUnpWTW1qZXlXK2tT?=
 =?utf-8?B?a2pXTFVzVnZlcVhEbVVJRmRBMzFtY3pobGlBdlg5TmI1ejE1MkQvdEhRVTNp?=
 =?utf-8?B?bFhLOWQ4aWpuRHN3R2xaeHhxNk44N2ZBaUE4b1NJa2hDVFNaTUtFdnF0T0JH?=
 =?utf-8?B?UlZESVV3aU41ZDN5ZEh6NFVBNmtQeDBXb04vdGl2Z3pqWVBuTWxPY1VEMzBx?=
 =?utf-8?B?N1FxdlVYU3M1bTFHSFcvRGRLemtOcjlSbDh2L0RrOVRXbm1tQlFPRERUZWpJ?=
 =?utf-8?B?blVJOE9hZnJhTjVQRnFXU3NBUmZ3aGoxWUlONU1yRDJ5bUhITGFoTTh0ckdU?=
 =?utf-8?B?aFgxQk9aMmRqcFJRdEpIRTJ4cEI5RGpBTjNrWHRZcFFTM2FPQiszaFpCbGdo?=
 =?utf-8?B?V01zMmN2MHcvU016R0tHeGJyZ2E3SmZHZDh4Vnd0YXNCcm1pMnlFNlRKb1pR?=
 =?utf-8?B?SmpuSzlwc1VVQ0l6VVo0UEZYMUtQc2xxbStMSG1QazJOMjhlazVRREpRT1kx?=
 =?utf-8?B?QzNEL3UxcmhUamxjYTFDUjJEYWJJWFc1S2g4UHI4SEtwVGtVYXBLZC9SUXF4?=
 =?utf-8?B?SUdyVEtQeXFXSUtyZmQ1UG9mRzdabEkrdThQM1JEYnQ3ZE5CTElQWDRmbDlB?=
 =?utf-8?B?SUoxM1JmQy9IZjlSb2g4dFkwSFJyQmtNZUJvTkQ5ZU5uV3lLUzZRcE5GVnB3?=
 =?utf-8?B?c2pua1orYjh2aCs4blJIU3VxZGRNWGRRQ2wvVS9TVGs3WmtCc3cxY2J1bGpK?=
 =?utf-8?B?czZySmZuM3hDdlpIc1E1WjFoZUVvOG51NW5TR01IT1NUdDdZU1ZTWVp1Vmpv?=
 =?utf-8?B?QWozSXBGTXFxWDZyRTd2QXpnRHVnSHRXL2xPZWJSYS8vblJHdkI1TlFWSW1m?=
 =?utf-8?B?aTR2MnA5ZjVaVWdmSFFPL3hEeitKU1g3cHo1Uk5FZHZyOU1rd211TzkvbHk5?=
 =?utf-8?B?eFh5ZkdBZzBkdkI2T2FieEUrdnJLOWlkWjJ6MTFBaWQ5eFVjY3pseDhqVVhu?=
 =?utf-8?Q?rtSWa0Nz0LHZKjkYAXH9f30G5?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0:
	 
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 0578a776-52ab-4883-b493-08dcf4437ab3
X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4313.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Oct 2024 15:49:44.0942
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: HiifhgVeNgbHpAE6RlQrLmOMD/j3R8GwwKDtjDftFSvh+LEAoSxhcBJ1yQC8WgjaAWDIzTEXi6pDq9kCzJ3Nsg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA2PR10MB4729
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1051,Hydra:6.0.680,FMLib:17.12.62.30
 definitions=2024-10-24_15,2024-10-24_02,2024-09-30_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 phishscore=0
 malwarescore=0 adultscore=0 mlxscore=0 mlxlogscore=958 spamscore=0
 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2409260000 definitions=main-2410240130
X-Proofpoint-GUID: 5SIsmPh7tVF5fnYofk1jThBAbmQMwdup
X-Proofpoint-ORIG-GUID: 5SIsmPh7tVF5fnYofk1jThBAbmQMwdup

Hi workqueue and scheduler maintainers,

As reported in 
https://lore.kernel.org/linux-fsdevel/df9db1ce-17d9-49f1-ab6d-7ed9a4f1f9c0@oracle.com/T/#m506b9edb1340cdddd87c6d14d20222ca8d7e8796, 
I am experiencing a workqueue lockup for v6.12-rcX.

At the point it occurs, the system becomes unresponsive and I cannot 
bring it back to life.

Enabling /proc/sys/kernel/softlockup_all_cpu_backtrace does not give 
anything extra in the way of debug. All I get is something like this:

Message from syslogd@jgarry-atomic-write-exp-e4-8-instance-20231214-1221 
at Oct 24 15:34:02 ...
  kernel:watchdog: BUG: soft lockup - CPU#29 stuck for 22s! [mysqld:14352]

Message from syslogd@jgarry-atomic-write-exp-e4-8-instance-20231214-1221 
at Oct 24 15:34:02 ...
  kernel:BUG: workqueue lockup - pool cpus=4 node=0 flags=0x0 nice=0 
stuck for 30s!

Message from syslogd@jgarry-atomic-write-exp-e4-8-instance-20231214-1221 
at Oct 24 15:34:02 ...
  kernel:BUG: workqueue lockup - pool cpus=31 node=0 flags=0x0 nice=0 
stuck for 49s!
^C

Can you advise on a robust method to get some debug from this system?

Maybe this is a scheduler issue, as Dave mentioned in that same thread.

Thanks,
John

From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5E8FB2101BC
	for <linux-kernel@vger.kernel.org>; Thu, 24 Oct 2024 18:56:17 +0000 (UTC)
Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201
ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116;
	t=1729796177; cv=none; b=ft2QBlqBQ1ygPCp8onKAunXaQmP6i4Am5uYdCCQA8pk/nxsOWp/NuqIfNDrhnwUKNaVaj/VGzgSg/GUto4IDdbVl24VpK6rca09WbuheJE37KLsI8EU9H0/UOMnIsPvU23VkBi+v0rMIfNHeXcMKk1aTaJKdhyJIzj8zzoYjhJc=
ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org;
	s=arc-20240116; t=1729796177; c=relaxed/simple;
	bh=E4XmmyC8mZy9ddoYetj7v9b0ZFybn3eTHJUa4mE18o8=;
	h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version:
	 Content-Type:Content-Disposition:In-Reply-To; b=B14VtJCPnmpVszAimBNP/TGNESR0lBtphLHZo+Qnvi0BOTbZemu8FRZNNirFRAVg0SJKFaAHDOn7mICm8GlYPqtN1SrfCGW4geWUsxu6R1CWQ2boLrZhEgmrG6ho9IHAy41KLxuuOF9IJ0JEjuw2DtpaQ9yBkoQsrl2L13JAiq8=
ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=UtWwph3O; arc=none smtp.client-ip=10.30.226.201
Authentication-Results: smtp.subspace.kernel.org;
	dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="UtWwph3O"
Received: by smtp.kernel.org (Postfix) with ESMTPSA id BEB20C4CECD;
	Thu, 24 Oct 2024 18:56:16 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org;
	s=k20201202; t=1729796176;
	bh=E4XmmyC8mZy9ddoYetj7v9b0ZFybn3eTHJUa4mE18o8=;
	h=Date:From:To:Cc:Subject:References:In-Reply-To:From;
	b=UtWwph3Ow9jiJR7pi1CpDbYrLm13Fhuqr3gx/4mG/dt7Xq3P5/jtswgl2N2ICimE/
	 488cJBHvDHCdVeat5S30Ee3t7eNXnM0F72mHgGd1TlF7eXi3uOdNnKdY4GSfgQKWYI
	 6/jXrDWfenxSR+cREJO0xRzngW5gy+0YIM7VTjIEFVWZPowEQrIPPYmsTi8Etlcuq2
	 6NpfmeDKi204LT2bwwYyFo+EPnUAaai22VhV1RgW29CfOPPHtEYQgEGeOtxkR46rYE
	 w1LRvBTc8aTs1ci+jkgckMq5w7RykxJ9XuWx+NwugDAJ5CRmyy0/ubw0SzWGXNg4Ze
	 zkmwwJPtpRFiw==
Date: Thu, 24 Oct 2024 08:56:15 -1000
From: Tejun Heo <tj@kernel.org>
To: John Garry <john.g.garry@oracle.com>
Cc: jiangshanlai@gmail.com, mingo@redhat.com, peterz@infradead.org,
	juri.lelli@redhat.com, jack@suse.cz, david@fromorbit.com,
	akpm@linux-foundation.org, linux-kernel@vger.kernel.org
Subject: Re: workqueue lockup debug
Message-ID: <ZxqYT9RfTBebgIIc@slm.duckdns.org>
References: <70cb224f-a468-40a0-82c7-735c590bdb43@oracle.com>
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <70cb224f-a468-40a0-82c7-735c590bdb43@oracle.com>

Hey, John.

On Thu, Oct 24, 2024 at 04:49:40PM +0100, John Garry wrote:
> Hi workqueue and scheduler maintainers,
> 
> As reported in https://lore.kernel.org/linux-fsdevel/df9db1ce-17d9-49f1-ab6d-7ed9a4f1f9c0@oracle.com/T/#m506b9edb1340cdddd87c6d14d20222ca8d7e8796,
> I am experiencing a workqueue lockup for v6.12-rcX.
> 
> At the point it occurs, the system becomes unresponsive and I cannot bring
> it back to life.
> 
> Enabling /proc/sys/kernel/softlockup_all_cpu_backtrace does not give
> anything extra in the way of debug. All I get is something like this:
> 
> Message from syslogd@jgarry-atomic-write-exp-e4-8-instance-20231214-1221 at
> Oct 24 15:34:02 ...
>  kernel:watchdog: BUG: soft lockup - CPU#29 stuck for 22s! [mysqld:14352]
> 
> Message from syslogd@jgarry-atomic-write-exp-e4-8-instance-20231214-1221 at
> Oct 24 15:34:02 ...
>  kernel:BUG: workqueue lockup - pool cpus=4 node=0 flags=0x0 nice=0 stuck
> for 30s!
> 
> Message from syslogd@jgarry-atomic-write-exp-e4-8-instance-20231214-1221 at
> Oct 24 15:34:02 ...
>  kernel:BUG: workqueue lockup - pool cpus=31 node=0 flags=0x0 nice=0 stuck
> for 49s!
> ^C
> 
> Can you advise on a robust method to get some debug from this system?
> 
> Maybe this is a scheduler issue, as Dave mentioned in that same thread.

So, while the symptom *could* be a workqueue bug, wq lockups are usually
more a side-effect of some other problem. All that wq is saying is that some
work item hasn't made forward progress for too long. Given that there are
soft lockup messages, which can't be caused by wq lockups but can definitely
trigger wq lockups, I'd start with debugging what's happening with the CPU
that triggered soft lockup. Soft lockups are nice in that they're
synchronous failures, so probably start with the backtrace?

Thanks.

-- 
tejun

From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id BC7D218BB9A
	for <linux-kernel@vger.kernel.org>; Fri, 25 Oct 2024 09:28:45 +0000 (UTC)
Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=205.220.177.32
ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116;
	t=1729848528; cv=fail; b=EgjAJyXpNHCItNoOo4MAD9JnFoCPcP/Kikffmb8q+N6sIyNoTX+aovsatng2Ko6ctacpiOIc5fL7aqbu0HfW43CUiT0t7Cw0+ZCfNrDIcB8TF04SHgKgu052C6MHqKkbuGnySjUAdT5e+MxpknhtjtnX8JjSE+DLUQ4WJ/vjM0A=
ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org;
	s=arc-20240116; t=1729848528; c=relaxed/simple;
	bh=WlP3IfP/wdwgH0loR2Icf1NnRaKgXIXXjDy/6dLIk/c=;
	h=Message-ID:Date:Subject:To:Cc:References:From:In-Reply-To:
	 Content-Type:MIME-Version; b=lKP3nvuem29SYO3XnCGmlStM15Kh6FJrFqW5LkgP3BKX81YMhgZ9uXiMfLttkL4dbhdVV8hE5GL1J4vm0alTP8/MIJbupAZnD5n9B5RT4EKpoFUfPaVFubS+Y+c9JzwHKOnK/MH+UYQ7POuvBMd46pTiQK7rV6hv66juMeXfZ3s=
ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b=kHyZDu6H; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b=pKPtfeUl; arc=fail smtp.client-ip=205.220.177.32
Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com
Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com
Authentication-Results: smtp.subspace.kernel.org;
	dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="kHyZDu6H";
	dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b="pKPtfeUl"
Received: from pps.filterd (m0246631.ppops.net [127.0.0.1])
	by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 49P8BXmu025031;
	Fri, 25 Oct 2024 09:28:32 GMT
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc
	:content-transfer-encoding:content-type:date:from:in-reply-to
	:message-id:mime-version:references:subject:to; s=
	corp-2023-11-20; bh=M7bkI8d98UEAlcwGaq/BHHcZoGoLS4PWU54MAquXhSg=; b=
	kHyZDu6HbViURQ18Zu+z2qUZn83idVl6xYu5d3eBtL+Y99CPcyCDUTiz2Sn3phQi
	uKtbfV7+Dnic5gbBD9lmk0fzPrkLhnFNDu8MrprqNAB09eFkShaiIamKq6ZTfNZ9
	g7By3h8DEg46O/fHD+IWZGz66s73F4U2fQj2YgjCNe2v04U3CUyxwhRyMMWsRcRk
	BCgw9sc5GwcPFHKHht+IfnG2ZzFi44OazOPxQDOITXgr4SPyu3hVOiBll3t608bZ
	5qBOAlCkc8NMAjUbzNj77zKXKag937i7UszDRelAa5OimGAIgQVqvuF07KSvjYP8
	AroBaQm94rtX1vDAzcolvw==
Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232])
	by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 42c57qmje0-1
	(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);
	Fri, 25 Oct 2024 09:28:32 +0000 (GMT)
Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1])
	by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 49P70nhZ025346;
	Fri, 25 Oct 2024 09:28:31 GMT
Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2174.outbound.protection.outlook.com [104.47.56.174])
	by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 42emhc2una-1
	(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);
	Fri, 25 Oct 2024 09:28:31 +0000
ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none;
 b=kGF+QnmdqZG7zg128wB8p50EJ9ZS2FCyzq92KxHwrB5VXImNPVszdj5HqocBS1DKYuHH76vjtYb9ElpmukYJSjtPEB3I3uL9rSPEmm2e7QiCaTyv4bxWq0PJY4Ve83CTnZzt/nPzlx0Ur3DCuwOfDii/4m92E1eCJYWCtvFfOfo18FKWOeU9Mg1zcm2w8mjIPHBdPXK2eYwRvQxO224APaOS9fWyszRIyLSfXMyaPTZPLznDpHXjTqwmDgVMzcbe26GW7pe5FC/O8Sghtpv8/K3XxVWp/ZC3GCuIh2kd/N1HoJ/BpGaYdb532311TNhQC5CMlRHS3rnW5qn0ynZZUw==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector10001;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=M7bkI8d98UEAlcwGaq/BHHcZoGoLS4PWU54MAquXhSg=;
 b=rQwun/edQ/DN2ZDkQEGieHvVbuQ9oBn97uzTCw/x2pDqYgylRdKXs+102m/fqoYwV1XbnXKF3Su1W4yOZU+jwhp3PkYoXcqiWbFoCBrwCNlwBBIh0503UG5L3k2GMhgp04blyaNN9a9jxywKLpvKp6HSzAxcxdOsmtSaH62KgzkmIOn/MjHITyGeY1vEAxycwm+uQxDDbDUSlEj7i3lQHncEIOweyDspJnoCCdNiQdNyl+rqGVJ2oIyCj7WRy06z9U+Qf/OT5u9YBiVKRmVldc7VNmkuH4VXt9DuG65Y+ICL7I98MVkkLS+YTAHVLEhtRZBdF5XQadTfOKFP+s2h5Q==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com;
 dkim=pass header.d=oracle.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=M7bkI8d98UEAlcwGaq/BHHcZoGoLS4PWU54MAquXhSg=;
 b=pKPtfeUly9VxTEHDwJa/7/IYJwhvhXZzUhEnUIzDVhWvwe3f/AJNxQISJDkVPCdl+tBtQq0whoznrDx1V3BGWCXBBftLDP2IWtkucDM68OSLy0NobOa/M+gOuT05XgWAwt+H8oLm9mA2WYDP/JfREaBvy+S5CVv9DrIvzfnT2lA=
Received: from DM6PR10MB4313.namprd10.prod.outlook.com (2603:10b6:5:212::20)
 by DS0PR10MB8128.namprd10.prod.outlook.com (2603:10b6:8:204::9) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8069.29; Fri, 25 Oct
 2024 09:28:28 +0000
Received: from DM6PR10MB4313.namprd10.prod.outlook.com
 ([fe80::4f45:f4ab:121:e088]) by DM6PR10MB4313.namprd10.prod.outlook.com
 ([fe80::4f45:f4ab:121:e088%5]) with mapi id 15.20.8093.018; Fri, 25 Oct 2024
 09:28:27 +0000
Message-ID: <9cfe9161-2ada-4fa0-bcdf-822401708fbf@oracle.com>
Date: Fri, 25 Oct 2024 10:28:24 +0100
User-Agent: Mozilla Thunderbird
Subject: Re: workqueue lockup debug
To: Tejun Heo <tj@kernel.org>
Cc: jiangshanlai@gmail.com, mingo@redhat.com, peterz@infradead.org,
        juri.lelli@redhat.com, jack@suse.cz, david@fromorbit.com,
        akpm@linux-foundation.org, linux-kernel@vger.kernel.org
References: <70cb224f-a468-40a0-82c7-735c590bdb43@oracle.com>
 <ZxqYT9RfTBebgIIc@slm.duckdns.org>
Content-Language: en-US
From: John Garry <john.g.garry@oracle.com>
Organization: Oracle Corporation
In-Reply-To: <ZxqYT9RfTBebgIIc@slm.duckdns.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-ClientProxiedBy: LO4P302CA0009.GBRP302.PROD.OUTLOOK.COM
 (2603:10a6:600:2c2::17) To DM6PR10MB4313.namprd10.prod.outlook.com
 (2603:10b6:5:212::20)
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-TrafficTypeDiagnostic: DM6PR10MB4313:EE_|DS0PR10MB8128:EE_
X-MS-Office365-Filtering-Correlation-Id: fbfe9ee3-791c-41fb-e44f-08dcf4d761bc
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|366016|1800799024;
X-Microsoft-Antispam-Message-Info:
	=?utf-8?B?UEFRT0ZGR0FzaTk0dnhuSERRck50WkNkSHN5RmxRZ0ZCL1FsVk1KUk9jQ09W?=
 =?utf-8?B?TmJPYmh2bml5dUtWL1JvTFFZZTJTalgzNWpIN1RkYkFSZU04L0xVUThyNGhw?=
 =?utf-8?B?ZS9Ud3Nya0pOdklwcDVTWkhQeGxwZ01URlNKREk0MjVQQ3NSc3NPT0tPa1c3?=
 =?utf-8?B?WWl0ZDBXVUNDTExOdWFZQ2t2UGNnb2dSTmFWWmQ5YWk1R1F1RSt0M29ldXRV?=
 =?utf-8?B?alRpODlpVnl2UmlHKzQ5NEdOMjA5OUJ3bzlmSGZaQWNxWXNQNXdYVnJtWkdL?=
 =?utf-8?B?V2ZEV241dmZlTkIvY0lDRmRKeVg2eGtSWDJGMS9pekh1VjlVUCs5MERjejl0?=
 =?utf-8?B?bDMxek5Iczh2U0p2YlBKR3JhOEFHRFJiTEdxWVFNS1YyMWxCL0FhS0FlUmpq?=
 =?utf-8?B?MmdTc2JpZ2ZmNTFDSzJ2NkQycTNpMnNVVGFnRG9IbENYOFdCNGk2bk9zMmhQ?=
 =?utf-8?B?bWtIWXBPM3k0T0MzbW1oRkp0L2ExOGc3bW1rd0l3Qzd1RlpoSmlOV2t6alZ3?=
 =?utf-8?B?R2FVbktzRlgzbHNDVis3c3pua2x1WVRqT0lLN3BTRjlNbHJXdk8ySVN3bUZG?=
 =?utf-8?B?RnJZQjE0aG95RDl3TVFWblhNM21BM3JWVk0wNDJqVU9TcDI5Ym5SdU1EeXRa?=
 =?utf-8?B?TmQzbytobzVWTll0TCtmNG5uMWw4aTJsd01YZ0pZOGY0cGpOSUU3YmRUbzZo?=
 =?utf-8?B?KzN6L0ZzRmM3a1hjOGhsTmlSby9VMWVITEdxUDROa21ZaFlMQTFGNXdWOGRK?=
 =?utf-8?B?cHRqbitHTHVEMFpjTEgrcnAwdmZTeUpFTnZiS2VxVXBYVERaSS92N3hKY29I?=
 =?utf-8?B?dWZhQTRaOWVxbTVObHdEdWZudFVkaGQrd3U3WGxmS3RwdGVHVEk2aTFjWFJO?=
 =?utf-8?B?ZWxqRHpLK1JETWEzVkVoc3RrQ3hLUklPMnB3cjVyL1g0RFR4TW1XRmI1S3pW?=
 =?utf-8?B?bTFJUjY4eUNUNW5aSm42N0tYUWNzMHJCeUNPRmhqMUpXT0dvYVdvV3Jlc3dW?=
 =?utf-8?B?REJXSStTOGpJanB4djQyazZmQVNZc2srUitVVTh0bmJkQ1RNb0JNQjZkNVoz?=
 =?utf-8?B?ZUlRL1NUYkE3RWppcXAyQjRmY0JMK3V3Y1NrRDR5VVpzNUZIaEs2bHlaaDA5?=
 =?utf-8?B?Y3N1VnFpRkZhTlZMcW13TWdDSlord29MSnpwRmFxZ1BvMlBEQVdjRnhOdHBq?=
 =?utf-8?B?ZEJHQkRsVjlleUM1MVZHU2dDQzhwM2h5bXBYU0ZoWEtYWXJwTFlxSWt0Zk9H?=
 =?utf-8?B?RnhEV1ZqZnU1VTBCR1czaDN2WVExdHVIb2FEZ2labzdSZWpPdk0rbUxSWHRN?=
 =?utf-8?B?UzFqV0tBclVWVUV0cDJSbEh3cjNqN2FMWmk0TXQxQUgydExGTVJreXJBOFJY?=
 =?utf-8?B?cnlkTzhVK1pmN2llTE5XUHR4eDlaMGNQMU91bC9lOEtMaEoxWEEzSjdPWmJM?=
 =?utf-8?B?czUrbGlFTHZsc2JXMC9yVGtqVkR1NnNlVWhMUTRRZVRFTzQ0aWxYbGg5dGRa?=
 =?utf-8?B?d0NOMGtPdjdBaHlDMXJCWEVvNGV1QXFGRUNnRUZJZ0ZCMlAwZ1VxQXF3cE9s?=
 =?utf-8?B?M0JKeFpUSVJ5ckJoUGIyMXEwNkFJOThYZjVrcTBhWjVvSytBMUFCS0QzSzFP?=
 =?utf-8?B?Wmw1TTR3YnIzdkJQMzBVQmRVUkJpb2I0V29YeUxadlIxSThJRW84RGlkZS9L?=
 =?utf-8?B?QW1USWNQSWNld1VlcGF0Mjc2a3k2Z1lMNm5xekZ1aEk0TW5lRXhRZytDQTFU?=
 =?utf-8?Q?3OcgOtT9P6PRx5cMbXX/X6aCcoht1/j1XnqPPVT?=
X-Forefront-Antispam-Report:
	CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4313.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(366016)(1800799024);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?NmhVUFZqZDdidUprRkRMUGNEMHh2SkxFOWxpVERJeVdmVHdEOSsyZGNmWGth?=
 =?utf-8?B?b2IrOU84ZlE4QUdabWg2ajBTOHJVQWUwODBRZFlHZ3hpTlp6SUN4UkttRDR5?=
 =?utf-8?B?Wm82eGJIN2F6MEJ2M2Y1Y2IvQng2Ry9USHN0ZUVDaXRqU3prZi8yNzBUTlRp?=
 =?utf-8?B?aDdUZ0tnRW0vRlB5S2d1QTduOGQ2U1lUOHkvNlI4WG8zMFN3QXlPYzJSclI0?=
 =?utf-8?B?Z0N2blV0aXZFMnEwcEZvMG5CSXZMTHN6QkNpUzh3ejg0eWJJYWkxdDNyUjRp?=
 =?utf-8?B?NUFKWTdYMTdVWmdCVW5OVktMc3lHaFFPMExwRW5pMjVUK1ROUnlBWUhwdWNx?=
 =?utf-8?B?bmhSaDQ2dlgzUTFPclplUk1XeFo0bkNjYWwxRDNSZ3BJaVZUeTEzMnRaa1Qv?=
 =?utf-8?B?TmR1amZTdjhiZG16MGhVZncxRW5jSE1mT2VxeFRxTTdGUVAvQWR2eUdmMUxn?=
 =?utf-8?B?YTJlTk53alFKL005eE5tK0N4ZW8zN0RPZVBUQUNoNnNIOGZ4WTBxc2ZYSG1x?=
 =?utf-8?B?d0lZSFo1T1I5RUYwamdNTFZPZ0xLN2hMenFXWldobldRMU1yL2c4NWxlTXF6?=
 =?utf-8?B?NHF3NDBXVzBlSTNQSENERHFET0tveWVRcSswNkYvT3d2Sk9RVkRPTWllSWNI?=
 =?utf-8?B?cTR5cEZ4U3hyektwWTZtY292VWFWeExuTHFCMUtBZWdDbUJaT29YdkxZcCtP?=
 =?utf-8?B?V1JwWThIU1RnK0ZKUFpybnpCclVIcWNkRHBaQVNmVzZiRHMzYmlwRzVhZXpZ?=
 =?utf-8?B?YkRWSzVkZ0g4VTQ4YkxkejNaRWJHWVVWUlgxRC9YbGpkQmIrZHhDMjZJMU9O?=
 =?utf-8?B?UjFNdmJyVXk4RW9mSk5kaVhsaWllVGRzcGVGYW8wVDdrVXU1VFpkWmxWOHJz?=
 =?utf-8?B?YzlQZ2lwcXFKUDV1UE5yRTZocXJwK3U3RW8zUWRMMkcvd2Y1OTF0QTdlT2Fn?=
 =?utf-8?B?cmlHR2grdFZHc3BKdnA3T0xGVVFiaFVCbitRTHVQVlFJZnBzYmd2M1FrR0to?=
 =?utf-8?B?cnNlS0hkQ2M3OXNZN0J1TTNVUngzVm15a0p5ajNFK25iQVcwQ3pxMzZXOWlU?=
 =?utf-8?B?SkNXcUduZlp2eXdmMncxcE13NEJXakxuajVkeGpVL3k2ZzU2a3Z3YXF2bDdp?=
 =?utf-8?B?WWtNNnB4bGJhYkE1a0lLR29VSG8zelhtS0ttQ2NYbnpMZjFoZ2E4Tm5qT3J5?=
 =?utf-8?B?QXdOajBKN1QvMEZyVWhyVVJOdjlDcmZWWjVyei9jZ25ST2JMdG9JWU44Q0E2?=
 =?utf-8?B?bmM3QWNLZWZPNlZ6R1I1WlhJQVBZTUdCbnR0NVBFTWVJekVHRTVudHF6Umpu?=
 =?utf-8?B?Sm1nRk1rdU5BUW4veFpZT0gxbHljdmNaTVRkdldTVGcveWFDM0pxbVN0QU5w?=
 =?utf-8?B?bzV6RFVKejJaVmkwVUo2cG0yck8zMmJDc3NtYlQ0aHI5MGEzdkllbEFCcGVr?=
 =?utf-8?B?dG45U1ZjbUh1eTMxR3hQNmdRNkVzamkzNWp2dXFxUVVkd0h1d05PbXZBT21j?=
 =?utf-8?B?cjZDVUtCbzEzajVGOTlLdUExSXFYYnBPM0ZMcmlDREFDSldyQlpHUng1aXRQ?=
 =?utf-8?B?dG14VkUyM2tUNTF5UVlta1k1QkUwYUx3cDY3RUFsUkgwVi9ISXRuOFE5aklv?=
 =?utf-8?B?VTNmUTlva3NXYXpvMDgyTmJBSXVTVEorekVuVFZ2YjNDeHNlYi9oRDNvUEph?=
 =?utf-8?B?Mk5YUkpsNGI4dS84Mk1CY2s2dVRKOVo2SWpwY09Ec0dxbzlZbnp3dWRIT09i?=
 =?utf-8?B?RWtyYnNWNkxPQVVHZm1WVVAxYnErZnVhNTVVQmUwWmNTQk1QaDhyZTlSeXNM?=
 =?utf-8?B?YzVLSlhGWmRoNDhpa3cvMGlSVGxvMHNRWm1tOWFBOUZrSFRoK0djUmFoQ1Fv?=
 =?utf-8?B?QVRPRUZxSlBNcWpEYWVUNHRnVG5MR3pSWE9KbDh3N0tMVGo3UVROblNVeFlT?=
 =?utf-8?B?eHIzTHRzdkt0b3gvNnpiQUw4REpEUG9RUVpPenVWbDZFWVgzekpEelJaSDc0?=
 =?utf-8?B?eEJGeDVNUGJQaEh0VnYvSnpmckZveDduc1lVQ3VZTHRhYVRFYUtpU1NVdVpa?=
 =?utf-8?B?N2JkTG1QN0cvYnF2aS9wQUxMVkNITzNmcnVsZjZsaDhjR0pJLzVLb1MwK1Za?=
 =?utf-8?B?NHN5Y09pQXlTTE9oQ0M3MkU3N25jQnJ6WDdUMFNaTzNreGFaRVErcW5jTUcv?=
 =?utf-8?B?WWc9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0:
	 
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: fbfe9ee3-791c-41fb-e44f-08dcf4d761bc
X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4313.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Oct 2024 09:28:27.8072
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Vjpyb9sn+N6V/xy/WB/fSlIou4CY+U7j1xEsWM7BusDBmZahcA1KdLtcPFnsFL+Ljq9LkypcYsh+LD86H1xt1g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB8128
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1051,Hydra:6.0.680,FMLib:17.12.62.30
 definitions=2024-10-25_06,2024-10-25_02,2024-09-30_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 spamscore=0 adultscore=0
 suspectscore=0 mlxscore=0 bulkscore=0 mlxlogscore=999 phishscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2409260000
 definitions=main-2410250073
X-Proofpoint-ORIG-GUID: _h6OCaW4S430Ur0X1FTNpdiYvBV8AD5o
X-Proofpoint-GUID: _h6OCaW4S430Ur0X1FTNpdiYvBV8AD5o

On 24/10/2024 19:56, Tejun Heo wrote:

Hi Tejun,

> 
> On Thu, Oct 24, 2024 at 04:49:40PM +0100, John Garry wrote:
>> Hi workqueue and scheduler maintainers,
>>
>> As reported in https://urldefense.com/v3/__https://lore.kernel.org/linux-fsdevel/df9db1ce-17d9-49f1-ab6d-7ed9a4f1f9c0@oracle.com/T/*m506b9edb1340cdddd87c6d14d20222ca8d7e8796__;Iw!!ACWV5N9M2RV99hQ!Jn5gFi_KIzLb-7uCOvHRIcf9Zd4RyHCTnm15Lo8u0KsgaMmwEPZHH2m4l-w4Fr-b6aRICoRWb5DU$ ,
>> I am experiencing a workqueue lockup for v6.12-rcX.
>>
>> At the point it occurs, the system becomes unresponsive and I cannot bring
>> it back to life.
>>
>> Enabling /proc/sys/kernel/softlockup_all_cpu_backtrace does not give
>> anything extra in the way of debug. All I get is something like this:
>>
>> Message from syslogd@jgarry-atomic-write-exp-e4-8-instance-20231214-1221 at
>> Oct 24 15:34:02 ...
>>   kernel:watchdog: BUG: soft lockup - CPU#29 stuck for 22s! [mysqld:14352]
>>
>> Message from syslogd@jgarry-atomic-write-exp-e4-8-instance-20231214-1221 at
>> Oct 24 15:34:02 ...
>>   kernel:BUG: workqueue lockup - pool cpus=4 node=0 flags=0x0 nice=0 stuck
>> for 30s!
>>
>> Message from syslogd@jgarry-atomic-write-exp-e4-8-instance-20231214-1221 at
>> Oct 24 15:34:02 ...
>>   kernel:BUG: workqueue lockup - pool cpus=31 node=0 flags=0x0 nice=0 stuck
>> for 49s!
>> ^C
>>
>> Can you advise on a robust method to get some debug from this system?
>>
>> Maybe this is a scheduler issue, as Dave mentioned in that same thread.
> 
> So, while the symptom *could* be a workqueue bug, wq lockups are usually
> more a side-effect of some other problem.

Sure

> All that wq is saying is that some
> work item hasn't made forward progress for too long. Given that there are
> soft lockup messages, which can't be caused by wq lockups but can definitely
> trigger wq lockups, I'd start with debugging what's happening with the CPU
> that triggered soft lockup. 

ok

> Soft lockups are nice in that they're
> synchronous failures, so probably start with the backtrace?
> 

I'm not getting any backtrace, which is not helping the issue.

But I'll continue to look.

Thanks,
John


From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id 56E7BFC0B
	for <linux-kernel@vger.kernel.org>; Tue, 29 Oct 2024 04:11:23 +0000 (UTC)
Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.170
ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116;
	t=1730175085; cv=none; b=tV2yz+NJ7bR6bH7j5GUBEzq9LugqgRyb9Q7tcSKWrxbbBmz+94nwFH34Yv9rsAB+lkm2BVKbRy/LmabdsHsbPDhSzM1L72QrPJI+z7L6dqxppI23CTPXUKCOWySv33wEEovu5V8Z61GvZor/XTitTa5Z87n1ESFgu0OhFHf7DKc=
ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org;
	s=arc-20240116; t=1730175085; c=relaxed/simple;
	bh=529W7VNks+B05G3MTaxORjcx3fS+jUYPlPca7CFLnrY=;
	h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version:
	 Content-Type:Content-Disposition:In-Reply-To; b=Z8e0FtjWHn5sw6U6JHa18Vvrs0kLK48V9PrHzeNIi15bOzJgm5Cy5W2V3V+oPoTTLeCUeR/4VIpQQXq1pcw3/tCMcxcfAKDfdCqWcWgH6zArSeQ0vlUSLZhmhvm2Ha2vnxDU1eWMBsNXLT6IspHM9kMGAxi+FDPCpjTcJnDZgIY=
ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=fromorbit.com; spf=pass smtp.mailfrom=fromorbit.com; dkim=pass (2048-bit key) header.d=fromorbit-com.20230601.gappssmtp.com header.i=@fromorbit-com.20230601.gappssmtp.com header.b=xuXQGIIP; arc=none smtp.client-ip=209.85.210.170
Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=fromorbit.com
Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fromorbit.com
Authentication-Results: smtp.subspace.kernel.org;
	dkim=pass (2048-bit key) header.d=fromorbit-com.20230601.gappssmtp.com header.i=@fromorbit-com.20230601.gappssmtp.com header.b="xuXQGIIP"
Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-71e5a1c9071so3829260b3a.0
        for <linux-kernel@vger.kernel.org>; Mon, 28 Oct 2024 21:11:23 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=fromorbit-com.20230601.gappssmtp.com; s=20230601; t=1730175082; x=1730779882; darn=vger.kernel.org;
        h=in-reply-to:content-disposition:mime-version:references:message-id
         :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to;
        bh=jfzW/XABIinOqwCPgFOuZcFxYdYSqcnrSwvREDZ46EU=;
        b=xuXQGIIPt8zmU9nPhjKDq1Hk29YqYs1deNDleZjVpAfaZ3abdLgi9TeybwG/1AKt/f
         5Nw8iyBPcuFCXG9TPW5k5buqhX98cahEWB9pYJvySG4d8fEWi1MziaCZtJcG9emn3GxV
         HCiiiqRtjK35iwPQIpBic77h9gj88xVasMmvmhdYN39UTxRLu3BgdtENOgu0FdGQxZlf
         v5obKAisjlGsGK1vqJGXQw4mNZ1uviyDzoamhah59GEGi2sXaok4pqOUyrMzsa05GTHz
         WGnxta17oUaqrcve0h0WBxz6h3HrTM019Z7CFg2pPrvkQOVnLKLn3PxJgL3NopoLcyWL
         bAaA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1730175082; x=1730779882;
        h=in-reply-to:content-disposition:mime-version:references:message-id
         :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date
         :message-id:reply-to;
        bh=jfzW/XABIinOqwCPgFOuZcFxYdYSqcnrSwvREDZ46EU=;
        b=t7Tg+GdIMIJf80u61Z6iRldkHFWGZYqBUNuAfgnQ3GxPcmTu4PW79yAQF0BawC/a3s
         f2fdRmtMuTv3jZk5TuZ1lWPPSLsQst/36/vODF80veJXhVrcuY3lr3IQJATQec+hkDcC
         yeflANlshocxVDLzH1Xgm7MN2gPWIKa/ZZX22AKO966INQUUdnul0k/kSyoj8NUODcDL
         M8sMCsQ+xck4o4r5ZJ5CZ3hZNqIx/B9DBe3gzM0PtHjqlCGVgZZb2Mo9ocO5V2zVc6S2
         XJynbbzo8t7Wt9sqvQxJ1l9Q5xbyqwLEC/05GwVc3k+U7C1Ix2MbraXG+V9lt4ONRgH/
         sr2A==
X-Forwarded-Encrypted: i=1; AJvYcCVLxEKvZdHTMZxV+8J4JDzlLF8S4EZzbXqYwq2aqCwNqvbMsj/wsKKFEvVlLzt3SJyFr2zxAuvOTAgNeHk=@vger.kernel.org
X-Gm-Message-State: AOJu0Yzk+6H1uTeRUG8xxE9Na+AmPIo6h8EHOn3hja5JcLujd1x30BYs
	LBeeR1PIrHuLUh8ARLLiTWCgnfQ8+6rPVZBE0N7aZpJ7TCOD9TPfXov80jezKug=
X-Google-Smtp-Source: AGHT+IGAj5JqIqR2Wtuz8giftudmpTFBTn4ufvaBw9vxzlmR5/XQUx0Ya49mrGfKy63tDpKaoKDAQA==
X-Received: by 2002:a05:6a20:9f8c:b0:1d9:20cf:2c24 with SMTP id adf61e73a8af0-1d9a8473c4amr15531541637.29.1730175082502;
        Mon, 28 Oct 2024 21:11:22 -0700 (PDT)
Received: from dread.disaster.area (pa49-186-86-168.pa.vic.optusnet.com.au. [49.186.86.168])
        by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72057a0b649sm6613415b3a.138.2024.10.28.21.11.22
        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
        Mon, 28 Oct 2024 21:11:22 -0700 (PDT)
Received: from dave by dread.disaster.area with local (Exim 4.96)
	(envelope-from <david@fromorbit.com>)
	id 1t5dZe-007Cse-2m;
	Tue, 29 Oct 2024 15:11:18 +1100
Date: Tue, 29 Oct 2024 15:11:18 +1100
From: Dave Chinner <david@fromorbit.com>
To: John Garry <john.g.garry@oracle.com>
Cc: tj@kernel.org, jiangshanlai@gmail.com, mingo@redhat.com,
	peterz@infradead.org, juri.lelli@redhat.com, jack@suse.cz,
	akpm@linux-foundation.org, linux-kernel@vger.kernel.org
Subject: Re: workqueue lockup debug
Message-ID: <ZyBgZr94Y9+xey4b@dread.disaster.area>
References: <70cb224f-a468-40a0-82c7-735c590bdb43@oracle.com>
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <70cb224f-a468-40a0-82c7-735c590bdb43@oracle.com>

On Thu, Oct 24, 2024 at 04:49:40PM +0100, John Garry wrote:
> Hi workqueue and scheduler maintainers,
> 
> As reported in https://lore.kernel.org/linux-fsdevel/df9db1ce-17d9-49f1-ab6d-7ed9a4f1f9c0@oracle.com/T/#m506b9edb1340cdddd87c6d14d20222ca8d7e8796,
> I am experiencing a workqueue lockup for v6.12-rcX.
> 
> At the point it occurs, the system becomes unresponsive and I cannot bring
> it back to life.
> 
> Enabling /proc/sys/kernel/softlockup_all_cpu_backtrace does not give
> anything extra in the way of debug. All I get is something like this:
> 
> Message from syslogd@jgarry-atomic-write-exp-e4-8-instance-20231214-1221 at
> Oct 24 15:34:02 ...
>  kernel:watchdog: BUG: soft lockup - CPU#29 stuck for 22s! [mysqld:14352]
> 
> Message from syslogd@jgarry-atomic-write-exp-e4-8-instance-20231214-1221 at
> Oct 24 15:34:02 ...
>  kernel:BUG: workqueue lockup - pool cpus=4 node=0 flags=0x0 nice=0 stuck
> for 30s!
> 
> Message from syslogd@jgarry-atomic-write-exp-e4-8-instance-20231214-1221 at
> Oct 24 15:34:02 ...
>  kernel:BUG: workqueue lockup - pool cpus=31 node=0 flags=0x0 nice=0 stuck
> for 49s!
> ^C
> 
> Can you advise on a robust method to get some debug from this system?
> 
> Maybe this is a scheduler issue, as Dave mentioned in that same thread.

I just got a new hit on the scheduler issue on 6.12-rc5:

[  172.477662] ------------[ cut here ]------------
[  172.480660] se->on_rq
[  172.480682] WARNING: CPU: 3 PID: 728318 at kernel/sched/fair.c:5629 pick_task_fair+0xb6/0x1b0
[  172.487172] Modules linked in:
[  172.488911] CPU: 3 UID: 0 PID: 728318 Comm: 291 Not tainted 6.12.0-rc5-dgc+ #273
[  172.492022] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[  172.496029] RIP: 0010:pick_task_fair+0xb6/0x1b0
[  172.497978] Code: 40 f3 f3 03 01 74 2a 41 80 7f 50 00 74 96 f6 05 31 f3 f3 03 01 75 8d c6 05 28 f3 f3 03 01 48 c7 c7 41 26 e4 82 e8 7a d7 fa ff <0f> 0b e9 73 ff ff ff c6 05 0d f3 f3 03 01 48 c7 c7 2f 26 e4 82 e8
[  172.505868] RSP: 0018:ffffc9002ec83d60 EFLAGS: 00010046
[  172.508111] RAX: bdcdd05ee831a400 RBX: 0000000000000000 RCX: 0000000000000027
[  172.511139] RDX: 0000000000000000 RSI: 00000000ffdfffff RDI: ffff88881fadc9c8
[  172.514326] RBP: ffffc9002ec83d88 R08: 00000000001fffff R09: ffff88a018a00000
[  172.517804] R10: 00000000005ffffd R11: 0000000000000004 R12: ffff888122b58c00
[  172.520788] R13: ffff88881faf0140 R14: ffff88881faf00c0 R15: ffff88819e8cd180
[  172.523748] FS:  0000000000000000(0000) GS:ffff88881fac0000(0000) knlGS:0000000000000000
[  172.527167] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  172.529571] CR2: 00007f378e8866fc CR3: 000000104daae000 CR4: 0000000000350ef0
[  172.532601] Call Trace:
[  172.533666]  <TASK>
[  172.534557]  ? show_regs+0x5e/0x70
[  172.535993]  ? __warn+0xd0/0x1d0
[  172.537430]  ? pick_task_fair+0xb6/0x1b0
[  172.539062]  ? report_bug+0x145/0x1f0
[  172.540619]  ? handle_bug+0x67/0x90
[  172.542060]  ? exc_invalid_op+0x1b/0x50
[  172.543854]  ? asm_exc_invalid_op+0x1b/0x20
[  172.545688]  ? pick_task_fair+0xb6/0x1b0
[  172.547353]  pick_next_task_fair+0x27/0x330
[  172.549090]  __schedule+0x2ad/0xb10
[  172.550460]  do_task_dead+0x43/0x50
[  172.551743]  do_exit+0x836/0xab0
[  172.552950]  do_group_exit+0x8f/0x90
[  172.554274]  __x64_sys_exit_group+0x17/0x20
[  172.555808]  x64_sys_call+0x2ed3/0x2ee0
[  172.557210]  do_syscall_64+0x68/0x130
[  172.558531]  ? exc_page_fault+0x62/0xc0
[  172.559907]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  172.561739] RIP: 0033:0x7f378e74b3c5
[  172.563036] Code: Unable to access opcode bytes at 0x7f378e74b39b.
[  172.565278] RSP: 002b:00007ffd21861488 EFLAGS: 00000206 ORIG_RAX: 00000000000000e7
[  172.568001] RAX: ffffffffffffffda RBX: 00007f378e857fa8 RCX: 00007f378e74b3c5
[  172.570589] RDX: 00000000000000e7 RSI: ffffffffffffff88 RDI: 0000000000000000
[  172.573182] RBP: 0000000000000000 R08: 00007ffd21861428 R09: 00007ffd218613af
[  172.575758] R10: 00007ffd21861320 R11: 0000000000000206 R12: 00007f378e856680
[  172.578356] R13: 00007f378e8b9f40 R14: 0000000000000001 R15: 00007f378e857fc0
[  172.580941]  </TASK>

There was nothing particularly obvious that caused it. It's a 64p
VM, running a heavy concurrent fstests workload that takes about 20
minutes to complete. There are a bit over 8000 tasks reported, most
of them kernel threads. The load is consuming about 50 CPUs, 40GB of
RAM, sustaining ~3GB/s of IO to/from disk across about 130 block
devices and the scheduler is context switching around 800,000 times
a second.

I have no idea how to reproduce this on demand - it randomly shows
up in about 1 in every 10 test runs and all the tests running at
that point in time report failure because they all detect this
warning in dmesg.

-Dave.
-- 
Dave Chinner
david@fromorbit.com

From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id C696920110A
	for <linux-kernel@vger.kernel.org>; Tue, 29 Oct 2024 08:08:23 +0000 (UTC)
Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=205.220.177.32
ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116;
	t=1730189306; cv=fail; b=o4DbamqHCZfNpeDPFidFFdOjjiAF8KgCmK7dMOGXnUfCyEZyqeFmZlvjseeKEL65OOsDnJ89a9jgIWDFWsNIaHI/gk4iAdYqDBwetWMfRjFU2h1u++vt5lIqaoBhKGDtGTi29I1/XhdTKW0P8P/i85cvJI0gCmwKDZD131DKZ4w=
ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org;
	s=arc-20240116; t=1730189306; c=relaxed/simple;
	bh=t4LpiQSQaE0AcnfbStRh1EGLYE6LXp8pwq5CQWFvcno=;
	h=Message-ID:Date:From:Subject:To:Cc:References:In-Reply-To:
	 Content-Type:MIME-Version; b=OfIXeHDXYy+RxLDDUfcg2KdaL+aYEGZ4vF5qCHCn0NDQ27Aj2WvWe869P8qGXceE/h0EzRLaa6CwVQ9T7AzMReqlyDW6FqHW29gVRg3E90zEVxsM/exIPDyCEON7n478a33bOVpIG5UHZ7iuH9iPn4//cOrWb6z3nj3m8y+VweU=
ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b=aOCFokzn; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b=czia+/GY; arc=fail smtp.client-ip=205.220.177.32
Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com
Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com
Authentication-Results: smtp.subspace.kernel.org;
	dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="aOCFokzn";
	dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b="czia+/GY"
Received: from pps.filterd (m0246630.ppops.net [127.0.0.1])
	by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 49T7tcbn014848;
	Tue, 29 Oct 2024 08:08:02 GMT
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc
	:content-transfer-encoding:content-type:date:from:in-reply-to
	:message-id:mime-version:references:subject:to; s=
	corp-2023-11-20; bh=3JWaDxxB/99ezk3bD26/LY+0Xyy7s/SHZCZKOVnHPgw=; b=
	aOCFokznN6mP3BFO4Yb6FtvtcnJt2kCrq5a44yPjEk1aSqElWexVSgGgQZqvy1w5
	r/s7uag11bP21W+B4SjDhqMQagc01rwQPFbHZK/SMkyGILviq8i6e1VFxW9Y6AN6
	I0ClQXqh7MoXyDUJyaFRPllwP2ziWA9UaEE4UPCAg6Ci9omWYNT5wQRKvpnEIdOM
	kPjeh74H5cWcaXNZyhXfxMVqLTGiMOeiz4vHf2Raq0Jn2Zfq+KECxxBzjeCKwr8L
	g+oV+fzIUHI+UW9VlqbCvmwpHVelheLe+TKc01o+9jO2sjksqiDOCToVfzgbwZPt
	iNEEOBLuoFmAI8a3qPBg1g==
Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20])
	by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 42grdqcpq9-1
	(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);
	Tue, 29 Oct 2024 08:08:02 +0000 (GMT)
Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1])
	by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 49T6al8Z034807;
	Tue, 29 Oct 2024 08:08:01 GMT
Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2173.outbound.protection.outlook.com [104.47.56.173])
	by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 42hnd7at7a-1
	(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);
	Tue, 29 Oct 2024 08:08:01 +0000
ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none;
 b=SklGcBqIb/B4fR3c7cY1fKSKtQQXXHjfs0K/oh0duYTrpDGyjq9gQzsxVO2fJJnrHS8KstvidI7LgXrqxqzbXjtAZjaQuNPiGrVUa+dycuIuEOP4v9Qp61xhFeqaeHGVVvuHrjLlR/USySfqOnTcvFS6I2TAvU+IqpDnX3WJKYb1vCsA1bFoT/hPfJlPToZlyP7VXqT/lJDp8ydoBT/JHZW1u5pDde9gBDlrTT0jOvGZRYNUi0g9iWzaN74Q9v1PxGhMweYz9AE+nTvWdaSQCMmht/YWfaY5wYI6fOei4iuyWtsUlXMVKKg6wUFTTrFO7/IyutEmbeXJPwwcl+R6Fw==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector10001;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=3JWaDxxB/99ezk3bD26/LY+0Xyy7s/SHZCZKOVnHPgw=;
 b=dfGHP4knZRAWa2IG3yI248tKQ2qz7svgN+iqtH2mDCiVrZnG3fzv+bAAPNHw5pYYq55whqN7C7cUllvM6FKQ8matHUdJ+TkIalNMu6idpA36S0bnRw1YSQNTwcUB6TQ/R2a1VfFeZzRN38EfBtBYd7MetxL24ewdCP3dTCnEk6KQYCfpBnf/e8jzYQmvCDP1PN9Vd2YVWf8g7PKrnLlHLORUpV5izdohiXfGYey+lprt5UVkVrxqTNwCbcUQpW9tSs9qwunz0OuC9EZVV76wMNTvnt0PUj9a/CrluNyaVBHTWlcpl8QJFK0CfjP/ye7pb57b/gkinXkTduiX4MFHnQ==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com;
 dkim=pass header.d=oracle.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=3JWaDxxB/99ezk3bD26/LY+0Xyy7s/SHZCZKOVnHPgw=;
 b=czia+/GYPenl7CAFp3OOQmMOl4yARA95NfVxxvJ4596rgYd+GZkdrYHKhEkpAYa1ObsdhQlmFu7bsZ8CstM7MFd15OxGeJxox9u+ZvAhOR2yl0dXmrc+5CZVZv7yCHV+39WF73EfhO2K6keRH9gDfE2BYbQFI6/KGWVXBq/9LOU=
Received: from DM6PR10MB4313.namprd10.prod.outlook.com (2603:10b6:5:212::20)
 by DS7PR10MB5184.namprd10.prod.outlook.com (2603:10b6:5:38e::20) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8093.24; Tue, 29 Oct
 2024 08:07:59 +0000
Received: from DM6PR10MB4313.namprd10.prod.outlook.com
 ([fe80::4f45:f4ab:121:e088]) by DM6PR10MB4313.namprd10.prod.outlook.com
 ([fe80::4f45:f4ab:121:e088%5]) with mapi id 15.20.8093.024; Tue, 29 Oct 2024
 08:07:59 +0000
Message-ID: <72dfc226-1a56-419b-bede-3be23ebffc35@oracle.com>
Date: Tue, 29 Oct 2024 08:07:55 +0000
User-Agent: Mozilla Thunderbird
From: John Garry <john.g.garry@oracle.com>
Subject: Re: workqueue lockup debug
To: Dave Chinner <david@fromorbit.com>
Cc: tj@kernel.org, jiangshanlai@gmail.com, mingo@redhat.com,
        peterz@infradead.org, juri.lelli@redhat.com, jack@suse.cz,
        akpm@linux-foundation.org, linux-kernel@vger.kernel.org
References: <70cb224f-a468-40a0-82c7-735c590bdb43@oracle.com>
 <ZyBgZr94Y9+xey4b@dread.disaster.area>
Content-Language: en-US
Organization: Oracle Corporation
In-Reply-To: <ZyBgZr94Y9+xey4b@dread.disaster.area>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-ClientProxiedBy: AS4P190CA0034.EURP190.PROD.OUTLOOK.COM
 (2603:10a6:20b:5d1::10) To DM6PR10MB4313.namprd10.prod.outlook.com
 (2603:10b6:5:212::20)
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-TrafficTypeDiagnostic: DM6PR10MB4313:EE_|DS7PR10MB5184:EE_
X-MS-Office365-Filtering-Correlation-Id: 2e0f6c02-6e89-4f73-7018-08dcf7f0cd3d
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|366016;
X-Microsoft-Antispam-Message-Info:
	=?utf-8?B?ODVCK00yU3RpUDh6UUMyZ25iWjV3VmpRNVBsYzNWSTRIc1ByVWliQkk4ZU9R?=
 =?utf-8?B?RWxHZThzcjV1WVU3VDkrbWc2dElLNk8xbHdLWEVNQzVRcUdXbi9LR3phVkto?=
 =?utf-8?B?NUxqNGtSTU11ZEZhdGV3UzFnQU0rOGZVWHduVzdSOWNNOG9yNU9OMG81eVUz?=
 =?utf-8?B?Vm91N09PUklUb2c5Y1YwbW9MeVR5WER3ZXV0MVJmd0I1bk1RUFR1WVN3Vng5?=
 =?utf-8?B?M0dGY3VUeitvWGNzTEFmQnZXbGt2c1hBc0I3RnZ5K2VNMVp2Y0RNWXBuNDNM?=
 =?utf-8?B?bTNyMjdRUVN2c0tqNjVHNUQ3V08waTJ2SzY0V1k0eXlRK3R5dUNuTHJiQ2ZQ?=
 =?utf-8?B?dFphQVJjNkNiOHA3M21LY0lBcnJINGg5RjUxanhSK3NPMDZtRVJ6ZElMSVlq?=
 =?utf-8?B?VU9ZYmV4eWxUWmNaVlUreUppZmFJS0UwWnM5YTNVQy91VGxRSHNRYnFjUlY0?=
 =?utf-8?B?dnBVU2Z3ZWZ1SXBic3RXd0t0a2h3L2VnNElPWTB0RFNJRVozZGhhSXpVVkx3?=
 =?utf-8?B?b2h1TURkTTZtMnByOHVIRkt6K0NNMWhUYXVqbExlMWZvVUZKUXA4STFTaUxF?=
 =?utf-8?B?V1ErQU9LRVlrSnpaTnBKbkVmRWZZRWVsMDJEYmlzdFhjenQyNzNvWG45Q2o4?=
 =?utf-8?B?SXlxYlNKMVNQSGgyamRhNTU2MHlrNkVrN2tEL3h1b1c3cVFza1AvMG5xamtZ?=
 =?utf-8?B?SjE2dGpaT1ZRNC9Xa09Yb0tQcHhEYjJoaG00VlVpbjNlbko0UnEzbHRXWkZT?=
 =?utf-8?B?a2c2VXg3Q293T2JiY2FIL0dBSDJJaGtlQWJIOGVaVnNKYXZuMUxwdkJXbzFV?=
 =?utf-8?B?eE9MMFhUbTlvL0dNWWVudENzQzcxWFFxTGp1bi9tKzAwUDVqVWZVdGc2WlFj?=
 =?utf-8?B?MkVvYjJXWlNoTitjelRoRGprUGl1ZUttRkpzNW4xNkIva3pmQ0Z2VFpIeWZu?=
 =?utf-8?B?dE12ZzlWaERxTnladEVWMStYTXVsbFQ5T3JpQktzbFdKZzRBSkx1MnVld0lw?=
 =?utf-8?B?cVpNLzZsVEo2K3ZvRjJ6L3llbGN6ZVM3Q201YmIrK2l1RWVDYzBGRDBlcHVU?=
 =?utf-8?B?WUdOUmVROGs2azg3MGlzanZLa0JvQUQ1OC9taWh6a2RFNFhFb1Q4YTgrNzVl?=
 =?utf-8?B?SnJWZkVDdHFUc3hPTjVadGV6WFU3bnkvcnlUQ0NqYjZERUlKY0VCT3JtN05M?=
 =?utf-8?B?Z2duNnROdkxUZlExZWJrK1RNcmRCTWplR0d6WXhHT0lHaFk5dWdYZU9Pa2xD?=
 =?utf-8?B?RjBVN1kwZzJOd1J1RjR0ckMxSURvSE9ZY2pIY2VWRzV5MXdtbEtmaHJuK3NY?=
 =?utf-8?B?NURuaGFaYjlHOVgycjVFZkc4RzR3bllGMXFWOGFUS2pkQ0prSE44U0dnYWds?=
 =?utf-8?B?V25BVlBtK0JKUGZFZXRNdm0rdDJNanMvclp3ZENlRkMvYkZlTE1zMll2Q3My?=
 =?utf-8?B?bkxHb0Y0eTkwZ3V3OWNsUlEvdnBvSVRrYzlkRlowaUlOczhIejJ3Rks3OGMr?=
 =?utf-8?B?SXJaOTErc1dTUXpmZ3NoWVA3OWgyVVVaR3dKVUxJV0hhK0RXaTFOMlR0ZEpp?=
 =?utf-8?B?ZGk2VU5lWGpCK2g3Q1Y5ZVIraXFkcXNMZktFSFpvMThRMmFYYTRhOHVSci9P?=
 =?utf-8?B?cGx5dm5mL3E5aXNuSHhBbzNBSWo3aTFkUml6Q0hhMXlQSzNJS0RSRjdPTjV2?=
 =?utf-8?B?WDM0ZGRyaFNoUnBVTDdWblhvVmM0bWZtTE9hQXFLVUZpTVkzd01wUEExY3ps?=
 =?utf-8?Q?/oWOdTtDkVz4yR7ImdgayrZlyMcK6/H/AiPqnJJ?=
X-Forefront-Antispam-Report:
	CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4313.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(366016);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?SjZGVHNZaTM1OVYwWTVGRHpzcGhsL2NKN1FrUFJnODd5NWdTVkxHNjhCa3Ru?=
 =?utf-8?B?YVVTWkhRZGorTjNOSUs1TGxxTnR4dUJ0QWVqeVJncnVwTjBKam4vU0pNdFdq?=
 =?utf-8?B?R1Z0KzFxUlFyei8vZUM4K0d4UGw2aExnYVZtQ0dqREpMNndMcWJkNVhzTmhF?=
 =?utf-8?B?bkNpc1ZVaE54L3QrdnJ3MEZqQnc2THdTWGlyRVBoZW0yU0U2RnplRnJQUVZv?=
 =?utf-8?B?VHlDeVdqOHIzVnZuZWNOVStNcy9rSURjTkprSDA5UEE0SXpHQmE5MFlhRW9w?=
 =?utf-8?B?Ti9FaTVkc01wZjZUcGZQcGUrRnlNaGhwcXFFU0UranJ0K2p0SEwxSm1hSnpp?=
 =?utf-8?B?cW9WMG5sd0JCdFpaVmNrTWVOUElzZVJVZVhUZHJrQlNmYUkrUlVXTU5GOU9C?=
 =?utf-8?B?dTRZZjF3Y013Wjl6NTRsdTlCejBKbHdYeDBXRjd4SEdQd0Evd2JWbHl1R05N?=
 =?utf-8?B?Z0dIYUxBYW5ReWEzVy9WZ3BhaC9oN05QdS9pY1I4Z2ZlWlJyMXFpdXA5eUFL?=
 =?utf-8?B?bURMYzAwMXdKeUdCNmo5YkV4SEliQWZJSVF0SjFXaHF6WGlGUnErb0V4bmhC?=
 =?utf-8?B?eStDZU1UVURQejFKNUpFSmxRVk51RnNuV0dwYXo0V0oxSXNVVWZYWGJBVWVK?=
 =?utf-8?B?R3d3ai83eWVaalBRZnprRDhmUmp1cEd5dzFHWVQ2YWdtR3EydkJwdXVucXJr?=
 =?utf-8?B?RXpYMm9CbG92azVOTFFIZnVVWXZkM0s2VWVlWWxhMHNrdWZNblMvbWZhVkJT?=
 =?utf-8?B?MTNMR0NuM1dHaHhlVlBvMytFbFY1VitOSlBzaXlUN1A5OU5SUnkrUU00Q1lv?=
 =?utf-8?B?YStBdEpjbkdOaTAwNEFoVzdTWnZGMmVuQ1d0bUFiUHV3UW4rODE0dzZTSzNP?=
 =?utf-8?B?bjR5c25DZ09GMk5nN3FZNFpLZE9JVDE2QnJ5VUJQWFV6K1B5cDJiNEIzNFZp?=
 =?utf-8?B?SXUrUVhsZkJrRjUyaWtuc2hTMnYrdXNaTFRkaG1mRkJjZVZ5UkdxUlVIVXB0?=
 =?utf-8?B?NHd0WHFvdTJRaFVJY0hJZTJPNU9jVHFTSUxaK09YSktDdFB5bDRkUGV6T3JF?=
 =?utf-8?B?cERNZTBQMGh6MWJ2OFkyM1lOZUY2eUY3cFRyNWJQNUhBMURnL0ZIOEJZbkRT?=
 =?utf-8?B?SGg4OERHYjVyNVlrMFNiZUZsVTI2SDN1eCtYaWpkempDWG9XcUxkUCtKQXpz?=
 =?utf-8?B?aEtHYnFJT205a0kyRllIaXl5TFVEL1J4aFljTTNaN3dld0tYSGJSc1hHWHNy?=
 =?utf-8?B?OEdPOWg1UUxGVlVOM05VdUozcDIwQ1JXYTB6SWNWSEdzeitvYm5FVTIxcXRI?=
 =?utf-8?B?OXYwTGVQY05yVGVZTnM5VjVYWEczaXV0c3N3VWNKZ0htWllGUmMxOUN1TnhS?=
 =?utf-8?B?aDJDaU1xSlRaR2RSM0N5cmI0YXBpYzU2U21kU2F2WHg2dFhpU0oweUNlQUJr?=
 =?utf-8?B?enVINVpWcGVYbklBSjBDUDc5U2JwZGwxRXJGRTlaM01xbk5pbDlSNE1sMlpP?=
 =?utf-8?B?Ymc3N2ZRSjJWS1NLMjRxMHdWWUp3Tmt4cDVsVXV3MWZ0VjFFbkozd2VHNVRl?=
 =?utf-8?B?cEE2TGJTa29IS3lVVnF6bHM0UG9QVW1laXBrMGFGRUdvdTJsK2t1b3ZMQXM5?=
 =?utf-8?B?bGtQa1FTajhWUitIcjZ4c0lFcUM4S05qVUVSamxRT0lXN0ZlR1NjZVk4OUQv?=
 =?utf-8?B?RlJwOWhEa0hoUEp6Si91eG8zN3ZYRzFsZmNrWW5CVzRMbDBoS2JRTEY1cWtZ?=
 =?utf-8?B?L1l2TVBkS2VpVk9xQXljbkk2RlExSVMrK01FZmkyQlRJYUw4bVgvYlUyRzV6?=
 =?utf-8?B?cC9jZ1ZNWDZicFFzbTBUUXBjdDN6VlNMck1KTkFFQ0pNN0xkVC9OTTFGTXdz?=
 =?utf-8?B?UnREazJNY3lRVjJ5QTlzNjZFc0x3TldPd2dFY0RSbUNVV1cyUEJPSE1yMDR1?=
 =?utf-8?B?RDA4WThtU3oyenk0dEplYVdlejBvV3R2b25xWmJuUVh4SFozN1BWMmRhZmtN?=
 =?utf-8?B?OXNjWThHNWpSRmM3TEFhMXZ2czhrV25sVVZMREo5Y1ZhcUR4aVBMR3BDNjBX?=
 =?utf-8?B?S1FIYXM4bWgwOGV1OGxmeFFvMkU1UWpvRnJsbGluRGI3K0xsUTd5am81bXZh?=
 =?utf-8?B?WnBOaW4ySWF3L1pkTjhmb3JMMTI4NTBnaHphRUR4UEltUEM1SGNDRElEakNH?=
 =?utf-8?B?cmc9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0:
	 
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2e0f6c02-6e89-4f73-7018-08dcf7f0cd3d
X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4313.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Oct 2024 08:07:58.9604
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: mUVSfQhP/SMLn+EHb8G6YoloBsinlUdMpHOt4Y4IbcwQR3ObP1Shkkpk/nTTlz1a79mfyx5fIJW9F+6Opwzw7A==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR10MB5184
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1051,Hydra:6.0.680,FMLib:17.12.62.30
 definitions=2024-10-29_04,2024-10-28_02,2024-09-30_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 malwarescore=0 spamscore=0
 phishscore=0 mlxscore=0 bulkscore=0 suspectscore=0 mlxlogscore=999
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2409260000
 definitions=main-2410290064
X-Proofpoint-GUID: ZLBCgUA2OnoFUjbVV10ZJns7l99r2HtZ
X-Proofpoint-ORIG-GUID: ZLBCgUA2OnoFUjbVV10ZJns7l99r2HtZ

On 29/10/2024 04:11, Dave Chinner wrote:
>> Can you advise on a robust method to get some debug from this system?
>>
>> Maybe this is a scheduler issue, as Dave mentioned in that same thread.
> I just got a new hit on the scheduler issue on 6.12-rc5:
> 
> [  172.477662] ------------[ cut here ]------------
> [  172.480660] se->on_rq
> [  172.480682] WARNING: CPU: 3 PID: 728318 at kernel/sched/fair.c:5629 pick_task_fair+0xb6/0x1b0
> [  172.487172] Modules linked in:
> [  172.488911] CPU: 3 UID: 0 PID: 728318 Comm: 291 Not tainted 6.12.0-rc5-dgc+ #273
> [  172.492022] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
> [  172.496029] RIP: 0010:pick_task_fair+0xb6/0x1b0
> [  172.497978] Code: 40 f3 f3 03 01 74 2a 41 80 7f 50 00 74 96 f6 05 31 f3 f3 03 01 75 8d c6 05 28 f3 f3 03 01 48 c7 c7 41 26 e4 82 e8 7a d7 fa ff <0f> 0b e9 73 ff ff ff c6 05 0d f3 f3 03 01 48 c7 c7 2f 26 e4 82 e8
> [  172.505868] RSP: 0018:ffffc9002ec83d60 EFLAGS: 00010046
> [  172.508111] RAX: bdcdd05ee831a400 RBX: 0000000000000000 RCX: 0000000000000027
> [  172.511139] RDX: 0000000000000000 RSI: 00000000ffdfffff RDI: ffff88881fadc9c8
> [  172.514326] RBP: ffffc9002ec83d88 R08: 00000000001fffff R09: ffff88a018a00000
> [  172.517804] R10: 00000000005ffffd R11: 0000000000000004 R12: ffff888122b58c00
> [  172.520788] R13: ffff88881faf0140 R14: ffff88881faf00c0 R15: ffff88819e8cd180
> [  172.523748] FS:  0000000000000000(0000) GS:ffff88881fac0000(0000) knlGS:0000000000000000
> [  172.527167] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  172.529571] CR2: 00007f378e8866fc CR3: 000000104daae000 CR4: 0000000000350ef0
> [  172.532601] Call Trace:
> [  172.533666]  <TASK>
> [  172.534557]  ? show_regs+0x5e/0x70
> [  172.535993]  ? __warn+0xd0/0x1d0
> [  172.537430]  ? pick_task_fair+0xb6/0x1b0
> [  172.539062]  ? report_bug+0x145/0x1f0
> [  172.540619]  ? handle_bug+0x67/0x90
> [  172.542060]  ? exc_invalid_op+0x1b/0x50
> [  172.543854]  ? asm_exc_invalid_op+0x1b/0x20
> [  172.545688]  ? pick_task_fair+0xb6/0x1b0
> [  172.547353]  pick_next_task_fair+0x27/0x330
> [  172.549090]  __schedule+0x2ad/0xb10
> [  172.550460]  do_task_dead+0x43/0x50
> [  172.551743]  do_exit+0x836/0xab0
> [  172.552950]  do_group_exit+0x8f/0x90
> [  172.554274]  __x64_sys_exit_group+0x17/0x20
> [  172.555808]  x64_sys_call+0x2ed3/0x2ee0
> [  172.557210]  do_syscall_64+0x68/0x130
> [  172.558531]  ? exc_page_fault+0x62/0xc0
> [  172.559907]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [  172.561739] RIP: 0033:0x7f378e74b3c5
> [  172.563036] Code: Unable to access opcode bytes at 0x7f378e74b39b.
> [  172.565278] RSP: 002b:00007ffd21861488 EFLAGS: 00000206 ORIG_RAX: 00000000000000e7
> [  172.568001] RAX: ffffffffffffffda RBX: 00007f378e857fa8 RCX: 00007f378e74b3c5
> [  172.570589] RDX: 00000000000000e7 RSI: ffffffffffffff88 RDI: 0000000000000000
> [  172.573182] RBP: 0000000000000000 R08: 00007ffd21861428 R09: 00007ffd218613af
> [  172.575758] R10: 00007ffd21861320 R11: 0000000000000206 R12: 00007f378e856680
> [  172.578356] R13: 00007f378e8b9f40 R14: 0000000000000001 R15: 00007f378e857fc0
> [  172.580941]  </TASK>
> 
> There was nothing particularly obvious that caused it. It's a 64p
> VM, running a heavy concurrent fstests workload that takes about 20
> minutes to complete. There are a bit over 8000 tasks reported, most
> of them kernel threads. The load is consuming about 50 CPUs, 40GB of
> RAM, sustaining ~3GB/s of IO to/from disk across about 130 block
> devices and the scheduler is context switching around 800,000 times
> a second.
> 
> I have no idea how to reproduce this on demand - it randomly shows
> up in about 1 in every 10 test runs and all the tests running at
> that point in time report failure because they all detect this
> warning in dmesg.

I have not been able to test v6.12-rc5 yet.

Can you share your config? And/Or compare to mine at 
https://pastebin.com/Y7DXnMG2

Maybe we can see what was merged for the sched in 6.12, and I can try 
before and after. I don't like the prospect of a full bisect.

Thanks,
John

From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id D4917802
	for <linux-kernel@vger.kernel.org>; Tue, 29 Oct 2024 15:51:56 +0000 (UTC)
Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=205.220.165.32
ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116;
	t=1730217119; cv=fail; b=owG6nWiPkJHJv7mFjzNSb9GCB2J2ejBjIAa2w2smNfSwXUJKWiZKRhDu2PUEw2uN+ds6r5FfgqIMkpljfffV5zJV/1mymDK2+js+XCngmuPLKhOaPjHZr20ONkbOFv6yaeFVF1bQxZc3yDNh2vwmeVRsASlOK+txjIMq4cFdDGw=
ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org;
	s=arc-20240116; t=1730217119; c=relaxed/simple;
	bh=QjSDItZkKh9mwo+/Au8up4zsOLDJRCv8Qx8jhU6IA/s=;
	h=Message-ID:Date:Subject:From:To:Cc:References:In-Reply-To:
	 Content-Type:MIME-Version; b=SuUl93viizwNbLN3M0JV7kdjhVgoR+/0wCszSLu/yS4vClffzUY9EEYZUgCGSMrXovwBqPjP1AVqUiQn+3HZDyvY9X/7F8POKAlpIt4/NSNYs/PNEa0RCykvwzfOb79daWtMIsGP3Arb4vNysDI4Qpt2Ti/UWATy+rEa3Y1dOQM=
ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b=PudTKI5R; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b=xD6ow4Gz; arc=fail smtp.client-ip=205.220.165.32
Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com
Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com
Authentication-Results: smtp.subspace.kernel.org;
	dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="PudTKI5R";
	dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b="xD6ow4Gz"
Received: from pps.filterd (m0246629.ppops.net [127.0.0.1])
	by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 49TEtfEY008663;
	Tue, 29 Oct 2024 15:51:38 GMT
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc
	:content-transfer-encoding:content-type:date:from:in-reply-to
	:message-id:mime-version:references:subject:to; s=
	corp-2023-11-20; bh=wNMUfW1/BnX/DFPwe2g2uSSS+3QO1PU0lua0Oipt4QU=; b=
	PudTKI5RdLAWJeXtyc7xPLliIg7iEKqs5sCnBhgNJL8IWGg/Ov1Jta3DQ9hD+utc
	qkxSU5fLj4uSCFs6mU8xa5GhUCOBHsvNvAXf0q9TEiheE95h2yfLZ8dFj1TWzcbV
	KSGi0F6/GyNuIOsKSE7OoEXVxAKSIrpXJVx0i5AH4MZlMk8SIijmiqDuSpWYXBgO
	h+FcKqiJVxEPAG7BF//s5lGZD143/3jKpjP5/H/MYxbG+mbrPQXRI42o8PKBLaY7
	xreFUyrhrPMilfKD98wry0hzSIR6DJoZ96tcNakRBqV0RC4latD3PwLRR/KQLRRE
	t5ViQFyi2lXrYFIl5xeWZg==
Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232])
	by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 42grgwdua9-1
	(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);
	Tue, 29 Oct 2024 15:51:38 +0000 (GMT)
Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1])
	by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 49TEcJDf040458;
	Tue, 29 Oct 2024 15:51:37 GMT
Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2170.outbound.protection.outlook.com [104.47.55.170])
	by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 42hnapatgj-1
	(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);
	Tue, 29 Oct 2024 15:51:37 +0000
ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none;
 b=WwP7TptJ9aR1kcyxzkrLbDPH2xvpXCvfBOndSdv8wY3L8JOnK9NFIoiwyZ1u2oc66HeUw6iMe3JE1de5PHjVIctZxMpjiwJhX8VkspJuCuPxisjL1x7JSUoP1GK7h5eGpZ2ZDD//6YJi1F6Xn2EmEX+be+O8vbgEUslp9xLKPKb6m8f6nTdnBJxDXMJ3EpfWY0jqacA5FwN/x9NOBDwwkglcBzWvQfuH31FWAH+6B0uTj+r6zhzCcXE/NYiRL4MR/MTcZaq5OhGdtmKFqvk7NekbteKmRfW5Negqjt1/fG+L8Ab0yITTLbxDtRBYIKcfwoSqHgARfXuFnHwCyhGl0Q==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector10001;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=wNMUfW1/BnX/DFPwe2g2uSSS+3QO1PU0lua0Oipt4QU=;
 b=hJ39Ai5WtsoHdDLwA2WoTrhd0eE+rHN30qAHT7zGVKwGcNmG5jc1HBmVYfXroPsQiP8sFgjJlbovF+lRRiIKg2X9Gez2vGrPLmoKJGhKRy2DSaPZ6i+h8GzzvBAfCzrYfm66GCy/T7QDj46hpRJvUAr+4p7hsvRCBHledzgIVj8GeaxZZVKrwJIWCi3w3yYorfpBecAR+pU1hakCb6srzEZ0Vx7jajhEjDyH8y96k8xM0y4cVz5y+LcFq/e2WR0ekNzE2f4VS1Cc3zEtkIrwOqK2ovzuiT9Jmv+BOpaYoWGX7PR/3Q2CY2JoadFeBXUiMr3IrOY+aYnZULnBAFqamA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com;
 dkim=pass header.d=oracle.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=wNMUfW1/BnX/DFPwe2g2uSSS+3QO1PU0lua0Oipt4QU=;
 b=xD6ow4GzfobNDwYD6Dr7wEf2MX/XnLRDCa8LhNNjn8+HsVuamzuSUCWFWcWc75Sdi0thMDU487p8BvYEEek4SRcGQnL4yx9msOtWs52oGhc3XzZ+h4jEUVB0v/BsI5PhQRgEOrW5MDLSi9Msx2cNLtiGFIV9Ao4WnFzK4uh62UM=
Received: from CH2PR10MB4312.namprd10.prod.outlook.com (2603:10b6:610:7b::9)
 by SN7PR10MB6382.namprd10.prod.outlook.com (2603:10b6:806:26e::20) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8093.32; Tue, 29 Oct
 2024 15:51:34 +0000
Received: from CH2PR10MB4312.namprd10.prod.outlook.com
 ([fe80::fd5e:682a:f1ac:d0a2]) by CH2PR10MB4312.namprd10.prod.outlook.com
 ([fe80::fd5e:682a:f1ac:d0a2%5]) with mapi id 15.20.8093.018; Tue, 29 Oct 2024
 15:51:33 +0000
Message-ID: <752118a5-96a4-4e60-bf3c-4e51e29be4b1@oracle.com>
Date: Tue, 29 Oct 2024 15:51:23 +0000
User-Agent: Mozilla Thunderbird
Subject: Re: workqueue lockup debug
From: John Garry <john.g.garry@oracle.com>
To: Dave Chinner <david@fromorbit.com>
Cc: tj@kernel.org, jiangshanlai@gmail.com, mingo@redhat.com,
        peterz@infradead.org, juri.lelli@redhat.com, jack@suse.cz,
        akpm@linux-foundation.org, linux-kernel@vger.kernel.org,
        chandan.babu@oracle.com
References: <70cb224f-a468-40a0-82c7-735c590bdb43@oracle.com>
 <ZyBgZr94Y9+xey4b@dread.disaster.area>
 <72dfc226-1a56-419b-bede-3be23ebffc35@oracle.com>
Content-Language: en-US
Organization: Oracle Corporation
In-Reply-To: <72dfc226-1a56-419b-bede-3be23ebffc35@oracle.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
X-ClientProxiedBy: SG3P274CA0010.SGPP274.PROD.OUTLOOK.COM (2603:1096:4:be::22)
 To CH2PR10MB4312.namprd10.prod.outlook.com (2603:10b6:610:7b::9)
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-TrafficTypeDiagnostic: CH2PR10MB4312:EE_|SN7PR10MB6382:EE_
X-MS-Office365-Filtering-Correlation-Id: d49374ba-7391-4ce8-ffc4-08dcf8318f8c
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|366016|1800799024;
X-Microsoft-Antispam-Message-Info:
	=?utf-8?B?SmgyeEw3VXFIR0VTd09kTUdCNDJMand1TTlrck1UOVpWRlR3bERRUnpCaFdn?=
 =?utf-8?B?b0R1TnE2bnhXWVpQb3B5SFNiZ1owaEoxTDExTy9pcFNmNHhJYXVMRHlHRkRZ?=
 =?utf-8?B?aUswdmdZQWNMTjlCb2JmcGJiNmVpNC9UL1NBYVdrcGtkR2MrK0crbDRxRVhh?=
 =?utf-8?B?cHdUcmtRUzE0NXJEY041RmpHd2wrd3VhZWhjb09nbVZhNFZwWmlnVytaTDNy?=
 =?utf-8?B?aTV6V210dml2Q3dlZXZYMVI0a1ZxUk45S0dVaGIveWtSa1llSHYwc1N3N01T?=
 =?utf-8?B?akhVTE9JQ2NVcXNDRzlQUDkrVHBPMGxzR21hSSsvS2tPWUJUNzJoVEw1ZnRF?=
 =?utf-8?B?TkNLS0dleWFySSttT0MxaUxHK3pIelJnS1dvV1U0bkZRNjR0MXA0VStmYlNr?=
 =?utf-8?B?SkZUTXQvbUdZQzltSWxEWkw0ZzM0TUJ3QlUvR2xGaFBBWjdBbzlSczl0dUcx?=
 =?utf-8?B?b1hpdi9sUlc0b3BZTDFoQkJoeitQNXhhTlhEQ3ZNWkM5amR0dHZvdnBDN1dS?=
 =?utf-8?B?b1J0eTlLbXh6WGpORHR1RHA1aThZMjZxWVgzc29sM281MnF0T1ZoaUxOdjk5?=
 =?utf-8?B?OVZWT0NQV0FQaU11bHNqWXFuWVd3emxpb1lhMklmZTgxelhaV2FVTmR0NVJm?=
 =?utf-8?B?WlEvd21UWG9EZ3pZL1owSFVwdmdhUEV1Q2dXY0NkNTU2WHNaZ0s0cjZIMElL?=
 =?utf-8?B?T1FIRWNEK043Nzdrb01SdWdhMWp4bHBHdkZOTEd3S3FqYjJtMnBSUTJoNEVP?=
 =?utf-8?B?YzYybmJCQWlwbzFNQ1UxYVAwTktMeFMvYklsYllXVndtRmoxZUs3US9nRENw?=
 =?utf-8?B?UHdKRFc4OUYxOExNd1ozQ2FCaXFvT0NpY08zY0xZVnVTRWxLeXYyQzFubHpE?=
 =?utf-8?B?a0gzdXBxcnBsQlhYZ0VFbTk0Q2lENWRKR1FETU5maHNPcEIvelpNbGdmZGha?=
 =?utf-8?B?UXpWTDVyUkxsUmtweTIwbmtHZVU5ZkV2VEF6MlBpdXVmbnpWM1o2bzdGR1RV?=
 =?utf-8?B?MUlCaEtRTVNqSXhvbjdrUWE1YWZhNlBGOURkS2lxcEdvSnZ0bHYzZnoxckxP?=
 =?utf-8?B?NVJWZDBBNklqRGpmVG9LR1cwN2JIdHAxTnIyTVRkaWkyWDF6LzEySFBnajBT?=
 =?utf-8?B?Tm1HWll6OFY1QUNHR0R0a29VeVR5bjRLdUJrc2xPMUt6Uk1kdG1pbmxMaS9W?=
 =?utf-8?B?eGdleHhLYVZiVzVCcWljbmVXYkIyaE9jVlR6c3VSZ2g3ZytXcjhzMUhSbGtw?=
 =?utf-8?B?aFR6Y1ZtRUNjWTA0ZDQwOVlZYU5OeTBLb2RVNFZ0OE1laitUWUF0V1dyM1hz?=
 =?utf-8?B?dVZ6c25NamU0VGlkSE80UDdlUlhtdVF4SVA1ejFVZlNzazhuN3BUU2lvL3dF?=
 =?utf-8?B?cTFjUjdKZ2pZL1RsVUxUZW5JcitXNzVnc1hvSm5IRFF3SWFiREQwR1JJY1o1?=
 =?utf-8?B?azQ0aU1NOVN6SmRBeXNhUXBKTEwrb2xFSU1mMkxaNjV1UWc3elNqcThncVFn?=
 =?utf-8?B?bVp4N2tRWDNFUkUrL0ZPNVVnVGxScFlBVm1zeU5TcjNjYnNvenoyUGJlRWdW?=
 =?utf-8?B?dTJkcmJOTFgrU0tQVm5rRll2MlpvZWsrTEloWGdUeWZmQlYvQjI0RW9uZkZT?=
 =?utf-8?B?OXBKeXJ2SmJndWVEcm1GamltU1Z5QVNPSVdMM0pHQjBzZXhtTndaU1JVMUFG?=
 =?utf-8?B?cEVDQkFmNUNab3piTEl5YlA3VHFocGM1R0FtWHd4SGtBSm1JU210NFl3PT0=?=
X-Forefront-Antispam-Report:
	CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH2PR10MB4312.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(366016)(1800799024);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?WjJOdHRlT2pKY0ZoL2tTeGl1SXRCSk9OQzdoZ2tJc1NSVTBHT2xtbUhHNGZm?=
 =?utf-8?B?WUF3cm8vZmZuZVFCdjhleC91M2dqYlB3NFk5Vk9qQ2N0dCtaYm9NdFNSaGNx?=
 =?utf-8?B?TWhvMVVkS3JPMVV1alU1Y3RocjBBcE44bmIwRkpjRURxVkNERmNCZjdVdTNH?=
 =?utf-8?B?UC80bm5MY2hPSVdTcjBoamU4eENRRjJMSW1HaUdHanFyd281Y0hTbGI2UkV1?=
 =?utf-8?B?emNFZTVJMlF0a0FwS25KeVlWMkRTSks1cWZtNW02alZEdEpjNzhxVjVWV21z?=
 =?utf-8?B?R2wxMVExcS9oNUZtNjZuTGtYNTg4U290ZWd6dm5LNnZEL2hBMlZYTXJxV1ZR?=
 =?utf-8?B?KzlHNlorN0xxbUlsNmhWVEpCNUNHV0NIa1Vwb2hjWjU0Y3VKNjNseUhmREJB?=
 =?utf-8?B?Q1BzeDJqQWF3cU5xR2g1MHhrVjd3M3VqbGJoa0hnZTZISms4NDM4S3Q5ME5S?=
 =?utf-8?B?VnpUakg0N1E3bmtQS1BVa0l3NHpIUVFIRTltSXc0N1F0K1dUY2FGMVNpUXd1?=
 =?utf-8?B?MVJvUHR6NWl5a041bzZ3dmg1QlBaY2thRDJ0SVZSMzhXQWF2T3EycXlDOFVy?=
 =?utf-8?B?WEFDUGNHdllXR210cGVkN3RaeUdCMEpDaHBWWSt1RkZOWXlHZEdSUERhc3FV?=
 =?utf-8?B?MTczZEZqaE1sZk5YQWcxZ1EyY05GMmpwUWl4d1laQ1VSSUt4MGNtSTc1N21k?=
 =?utf-8?B?bWNFdWhSTHVWOXFRMUdzcVJHMjJkWTA4TkdEVjhKVHF4VU1zNVh6V3UzVTJU?=
 =?utf-8?B?emJSanNZYVVJRGNNQjdrU2VLdTJTZHN3SjREZTl6RlZDWWVZSkdNUW9YZ0Jw?=
 =?utf-8?B?Y2gvVTd0Q1JieC9vODMyWHk1bXlMcDA2cWQrQjlEQ0xjRjdUTWNVdUtVSE0r?=
 =?utf-8?B?SDBBeHdFdk9JaG80RENqVXlEYThDMmlWaXJ3VURUd29CVyt5S0pHYnVSOEd1?=
 =?utf-8?B?aS9IenBqd0VPQlR2bDJMeGZ3ZVlFYmJZeVVTM3ExckVLbThyWmprZENSdnJs?=
 =?utf-8?B?am9VL1ZIZ0tLeHBvSjNOcUtKTE0rTVJzcDFCaU9ZK2ExNUZiR0Q0cldaNlBP?=
 =?utf-8?B?NjF6aEtnS0Npa3AxRUNQS0hweTZwL3hBZS9CVXhCcitPeHVEWUtweEE3eFho?=
 =?utf-8?B?SXlOVjFpTnJIZUVnUjdFNHdIZ1hIb2ZVU0N3Vm8zUEtnQU9pNW1ITDd6RmlP?=
 =?utf-8?B?a2FoM25SNHRIazNqcTFtOGdiK2YzMEFUSzJ2SS9mL1Y2ZXVrcW1rS1lha3Ft?=
 =?utf-8?B?c2VBRVNLcjFoUkJBZ0ZQTmFWRm1EeTR6OW9DNWcva2tVdm5sMGxKd2YxOHFs?=
 =?utf-8?B?ZzRTVjg2OUJaaHc2a0p1ZS9uQmRaNm5kZUFQWDVmcXdTcC9KM3RvNy9jWlFh?=
 =?utf-8?B?bEUwQmdSRXFGZGhtYUZPeFdmdHluYW5LUzRJMEw4OFNlR2g4a05aMWVvR0Z4?=
 =?utf-8?B?QlY3V0RKWGJ1N1MwdzVoNUlKNUJhM3lUeXd6bGEyQXRYMFVjTHFHbTlsVlBh?=
 =?utf-8?B?dXF6ZWZBVlJlcnNuTnM4WFB2MGJiVTFIeXc2Tm5pNWNqdDlRZlk2MzhoVWFi?=
 =?utf-8?B?YUJBTWxieXYvbW4rV1NsUWIzbmNYdmNlRmx4MjZISlBxRlBHSkNvM3J5VUoz?=
 =?utf-8?B?eUQ2OFRRYWVRdEc2RXY3S2h1V2xOMm5xU3dQS21MWXNqUTB4MUk2Wkt3ejdX?=
 =?utf-8?B?RWUwTW12TzJIMzZjQWRhdTBVaXk4LzVBLzhOZUU5dmpGWjFXeVNaU0g4bGRi?=
 =?utf-8?B?SzhEdG1KRmRBeGowQm9PckZienpCTi9wZHA3UWRGSmtsU1pHMlBCRVIxZHdJ?=
 =?utf-8?B?Y1IvaVl0blh3QWdJZXNjMEpmSDMwY3o3cVBDcHhpM2hKMElhZmhHbnRxdUdq?=
 =?utf-8?B?Q3UxQXNzQ1NmZFFXQVp0NEM3WmdOYnl2MnpuaC9xSVhDRTh4bUVjSGNFRFlI?=
 =?utf-8?B?RlVPZFE4TjhjcUZYUnJjdW9ReVVEMzJ2RjhJV1NKTld0d2E0ekVLdk1zbC9q?=
 =?utf-8?B?Z0NLK3RCWkZ4WFh0TS91RmhlbDUyL1RMTXh4SUJPNWZCS3A0NlRHUDZVZ3RZ?=
 =?utf-8?B?dVpvSHRZRnpJa3hMWTg2OXZ5YTJ4aXlZZW5jNDZRdUM4NjRKRFBhTFdMUzRw?=
 =?utf-8?B?Z1VWL291VXhGei9nVVBmTVBiU05JeVlnQTJFOHpYQzgzcEhxZFpEbmNaUEpV?=
 =?utf-8?B?RHc9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0:
	 
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d49374ba-7391-4ce8-ffc4-08dcf8318f8c
X-MS-Exchange-CrossTenant-AuthSource: CH2PR10MB4312.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Oct 2024 15:51:32.9036
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: er1SLk5m/9MZtuLaPT+q+m6mt3e3Npj+yut3N+HxcrB7nzawAcsqKWLevl6ep29q8Rp3ftHdLV+RNlMl0s3o7Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR10MB6382
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1051,Hydra:6.0.680,FMLib:17.12.62.30
 definitions=2024-10-29_11,2024-10-29_01,2024-09-30_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxscore=0 adultscore=0
 mlxlogscore=999 phishscore=0 bulkscore=0 malwarescore=0 spamscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2409260000
 definitions=main-2410290121
X-Proofpoint-ORIG-GUID: Bvxgg417I21rTYhOnLgzY1Fq-Pop3Xyi
X-Proofpoint-GUID: Bvxgg417I21rTYhOnLgzY1Fq-Pop3Xyi

On 29/10/2024 08:07, John Garry wrote:

+ Chandan

Note that Chandan also reported this other issue 
https://lore.kernel.org/all/87r08zvsdd.fsf@debian-BULLSEYE-live-builder-AMD64/, 
which still exists in rc4

>>> Maybe this is a scheduler issue, as Dave mentioned in that same thread.
>> I just got a new hit on the scheduler issue on 6.12-rc5:
>>
>> [  172.477662] ------------[ cut here ]------------
>> [  172.480660] se->on_rq
>> [  172.480682] WARNING: CPU: 3 PID: 728318 at kernel/sched/fair.c:5629 

That WARN is new code in 6.12

>> pick_task_fair+0xb6/0x1b0
>> [  172.487172] Modules linked in:
>> [  172.488911] CPU: 3 UID: 0 PID: 728318 Comm: 291 Not tainted 6.12.0- 
>> rc5-dgc+ #273
>> [  172.492022] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), 
>> BIOS 1.16.3-debian-1.16.3-2 04/01/2014
>> [  172.496029] RIP: 0010:pick_task_fair+0xb6/0x1b0
>> [  172.497978] Code: 40 f3 f3 03 01 74 2a 41 80 7f 50 00 74 96 f6 05 
>> 31 f3 f3 03 01 75 8d c6 05 28 f3 f3 03 01 48 c7 c7 41 26 e4 82 e8 7a 
>> d7 fa ff <0f> 0b e9 73 ff ff ff c6 05 0d f3 f3 03 01 48 c7 c7 2f 26 e4 
>> 82 e8
>> [  172.505868] RSP: 0018:ffffc9002ec83d60 EFLAGS: 00010046
>> [  172.508111] RAX: bdcdd05ee831a400 RBX: 0000000000000000 RCX: 
>> 0000000000000027
>> [  172.511139] RDX: 0000000000000000 RSI: 00000000ffdfffff RDI: 
>> ffff88881fadc9c8
>> [  172.514326] RBP: ffffc9002ec83d88 R08: 00000000001fffff R09: 
>> ffff88a018a00000
>> [  172.517804] R10: 00000000005ffffd R11: 0000000000000004 R12: 
>> ffff888122b58c00
>> [  172.520788] R13: ffff88881faf0140 R14: ffff88881faf00c0 R15: 
>> ffff88819e8cd180
>> [  172.523748] FS:  0000000000000000(0000) GS:ffff88881fac0000(0000) 
>> knlGS:0000000000000000
>> [  172.527167] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [  172.529571] CR2: 00007f378e8866fc CR3: 000000104daae000 CR4: 
>> 0000000000350ef0
>> [  172.532601] Call Trace:
>> [  172.533666]  <TASK>
>> [  172.534557]  ? show_regs+0x5e/0x70
>> [  172.535993]  ? __warn+0xd0/0x1d0
>> [  172.537430]  ? pick_task_fair+0xb6/0x1b0
>> [  172.539062]  ? report_bug+0x145/0x1f0
>> [  172.540619]  ? handle_bug+0x67/0x90
>> [  172.542060]  ? exc_invalid_op+0x1b/0x50
>> [  172.543854]  ? asm_exc_invalid_op+0x1b/0x20
>> [  172.545688]  ? pick_task_fair+0xb6/0x1b0
>> [  172.547353]  pick_next_task_fair+0x27/0x330
>> [  172.549090]  __schedule+0x2ad/0xb10
>> [  172.550460]  do_task_dead+0x43/0x50
>> [  172.551743]  do_exit+0x836/0xab0
>> [  172.552950]  do_group_exit+0x8f/0x90
>> [  172.554274]  __x64_sys_exit_group+0x17/0x20
>> [  172.555808]  x64_sys_call+0x2ed3/0x2ee0
>> [  172.557210]  do_syscall_64+0x68/0x130
>> [  172.558531]  ? exc_page_fault+0x62/0xc0
>> [  172.559907]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
>> [  172.561739] RIP: 0033:0x7f378e74b3c5
>> [  172.563036] Code: Unable to access opcode bytes at 0x7f378e74b39b.
>> [  172.565278] RSP: 002b:00007ffd21861488 EFLAGS: 00000206 ORIG_RAX: 
>> 00000000000000e7
>> [  172.568001] RAX: ffffffffffffffda RBX: 00007f378e857fa8 RCX: 
>> 00007f378e74b3c5
>> [  172.570589] RDX: 00000000000000e7 RSI: ffffffffffffff88 RDI: 
>> 0000000000000000
>> [  172.573182] RBP: 0000000000000000 R08: 00007ffd21861428 R09: 
>> 00007ffd218613af
>> [  172.575758] R10: 00007ffd21861320 R11: 0000000000000206 R12: 
>> 00007f378e856680
>> [  172.578356] R13: 00007f378e8b9f40 R14: 0000000000000001 R15: 
>> 00007f378e857fc0
>> [  172.580941]  </TASK>
>>
>> There was nothing particularly obvious that caused it. It's a 64p
>> VM, running a heavy concurrent fstests workload that takes about 20
>> minutes to complete. There are a bit over 8000 tasks reported, most
>> of them kernel threads. The load is consuming about 50 CPUs, 40GB of
>> RAM, sustaining ~3GB/s of IO to/from disk across about 130 block
>> devices and the scheduler is context switching around 800,000 times
>> a second.
>>
>> I have no idea how to reproduce this on demand - it randomly shows
>> up in about 1 in every 10 test runs and all the tests running at
>> that point in time report failure because they all detect this
>> warning in dmesg.
> 
> I have not been able to test v6.12-rc5 yet.

Just to confirm, v6.12-rc5 also hangs for me

> 
> Can you share your config? And/Or compare to mine at https:// 
> pastebin.com/Y7DXnMG2
> 
> Maybe we can see what was merged for the sched in 6.12, and I can try 
> before and after. I don't like the prospect of a full bisect.


From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from wp530.webpack.hosteurope.de (wp530.webpack.hosteurope.de [80.237.130.52])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id 493F11DB534
	for <linux-kernel@vger.kernel.org>; Thu,  7 Nov 2024 12:39:48 +0000 (UTC)
Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=80.237.130.52
ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116;
	t=1730983191; cv=none; b=pTHGlGzpCYVZUJmXSpENLLZfAHmENqGvDW9ATtNGptlsyYVYTRJXuMaCGJCGT8redwC/4PRH1peyjwANHu3rSEXS1ko/TXdFDPW44UonGmsjezLnr8tsa+qDXHvyfq3JOVyP5zM7awtxchCFOnxbmVgkhW8ROk/qPChCkkoRXHA=
ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org;
	s=arc-20240116; t=1730983191; c=relaxed/simple;
	bh=l6ufvF85HwoJWUhmgZ2iuzTo2qL0SCVeIkLSb687duA=;
	h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From:
	 In-Reply-To:Content-Type; b=CxFB3hTsyVi6jiQIqhEVwaseawcRjVYk52ha2GlFU/3XBwsGpV2hFbQJ0zfR/fRAt7wgluBCX0zuvan8eqc21C+pwVw2jeL01oi5cwH9Gx96cy2avjfRLck/rQyljr3o0aBMwml+lEBIWi3A3LJfMQZKugSfiYZn2wXIvx+xUsI=
ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=leemhuis.info; spf=pass smtp.mailfrom=leemhuis.info; dkim=pass (2048-bit key) header.d=leemhuis.info header.i=@leemhuis.info header.b=ZDKFiOIy; arc=none smtp.client-ip=80.237.130.52
Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=leemhuis.info
Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=leemhuis.info
Authentication-Results: smtp.subspace.kernel.org;
	dkim=pass (2048-bit key) header.d=leemhuis.info header.i=@leemhuis.info header.b="ZDKFiOIy"
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
	d=leemhuis.info; s=he214686; h=Content-Transfer-Encoding:Content-Type:
	In-Reply-To:From:References:Cc:To:Subject:MIME-Version:Date:Message-ID:From:
	Sender:Reply-To:Subject:Date:Message-ID:To:Cc:MIME-Version:Content-Type:
	Content-Transfer-Encoding:Content-ID:Content-Description:In-Reply-To:
	References; bh=E+fHOiPBBxWdnVVpaSlXJefWhHOYuF9e5zPJ6Yd0KAs=; t=1730983189;
	x=1731415189; b=ZDKFiOIycKUwwCzz2mVC6lI42NW4cUDs0GtctCBlvKWp614sIxIhhhUr9hOUs
	kg/zrW9MJwwnuyvmKnNudoXVoqhIHr/nQGDnlvLB70R1VZ/hI7GrHw1w317CUEbp36gKrH1vt3dtH
	i4rOhBDoz0Mh70+lvpkf3M5hAru5vRWfusYT/354jg+7+kGXO+rNc83TToX7Kq/o+C3gmk5lyxqJJ
	QFFS/ZI+QGBGfXJW+fLRAOS+Q6LGKTf/lNHeaaBY5EAZHk917EbQO4aBNGfauRb47ThpX8VSWLlCk
	pjKV5E8AZlLEEZz+a9hZpAFgw0IV1vtKD/tOM3LY/WZhfy7i9A==;
Received: from [2a02:8108:8980:2478:87e9:6c79:5f84:367d]; authenticated
	by wp530.webpack.hosteurope.de running ExIM with esmtpsa (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128)
	id 1t91nY-00009W-BT; Thu, 07 Nov 2024 13:39:40 +0100
Message-ID: <4e58d34a-ce45-437a-95a2-3ba21f35bbb5@leemhuis.info>
Date: Thu, 7 Nov 2024 13:39:39 +0100
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: workqueue lockup debug
To: John Garry <john.g.garry@oracle.com>, tj@kernel.org,
 jiangshanlai@gmail.com, mingo@redhat.com, peterz@infradead.org,
 juri.lelli@redhat.com
Cc: jack@suse.cz, david@fromorbit.com, akpm@linux-foundation.org,
 linux-kernel@vger.kernel.org
References: <70cb224f-a468-40a0-82c7-735c590bdb43@oracle.com>
From: Thorsten Leemhuis <regressions@leemhuis.info>
Content-Language: en-MW
Autocrypt: addr=linux@leemhuis.info; keydata=
 xsFNBFJ4AQ0BEADCz16x4kl/YGBegAsYXJMjFRi3QOr2YMmcNuu1fdsi3XnM+xMRaukWby47
 JcsZYLDKRHTQ/Lalw9L1HI3NRwK+9ayjg31wFdekgsuPbu4x5RGDIfyNpd378Upa8SUmvHik
 apCnzsxPTEE4Z2KUxBIwTvg+snEjgZ03EIQEi5cKmnlaUynNqv3xaGstx5jMCEnR2X54rH8j
 QPvo2l5/79Po58f6DhxV2RrOrOjQIQcPZ6kUqwLi6EQOi92NS9Uy6jbZcrMqPIRqJZ/tTKIR
 OLWsEjNrc3PMcve+NmORiEgLFclN8kHbPl1tLo4M5jN9xmsa0OZv3M0katqW8kC1hzR7mhz+
 Rv4MgnbkPDDO086HjQBlS6Zzo49fQB2JErs5nZ0mwkqlETu6emhxneAMcc67+ZtTeUj54K2y
 Iu8kk6ghaUAfgMqkdIzeSfhO8eURMhvwzSpsqhUs7pIj4u0TPN8OFAvxE/3adoUwMaB+/plk
 sNe9RsHHPV+7LGADZ6OzOWWftk34QLTVTcz02bGyxLNIkhY+vIJpZWX9UrfGdHSiyYThHCIy
 /dLz95b9EG+1tbCIyNynr9TjIOmtLOk7ssB3kL3XQGgmdQ+rJ3zckJUQapLKP2YfBi+8P1iP
 rKkYtbWk0u/FmCbxcBA31KqXQZoR4cd1PJ1PDCe7/DxeoYMVuwARAQABzSdUaG9yc3RlbiBM
 ZWVtaHVpcyA8bGludXhAbGVlbWh1aXMuaW5mbz7CwZQEEwEKAD4CGwMFCwkIBwMFFQoJCAsF
 FgIDAQACHgECF4AWIQSoq8a+lZZX4oPULXVytubvTFg9LQUCX31PIwUJFmtPkwAKCRBytubv
 TFg9LWsyD/4t3g4i2YVp8RoKAcOut0AZ7/uLSqlm8Jcbb+LeeuzjY9T3mQ4ZX8cybc1jRlsL
 JMYL8GD3a53/+bXCDdk2HhQKUwBJ9PUDbfWa2E/pnqeJeX6naLn1LtMJ78G9gPeG81dX5Yq+
 g/2bLXyWefpejlaefaM0GviCt00kG4R/mJJpHPKIPxPbOPY2REzWPoHXJpi7vTOA2R8HrFg/
 QJbnA25W55DzoxlRb/nGZYG4iQ+2Eplkweq3s3tN88MxzNpsxZp475RmzgcmQpUtKND7Pw+8
 zTDPmEzkHcUChMEmrhgWc2OCuAu3/ezsw7RnWV0k9Pl5AGROaDqvARUtopQ3yEDAdV6eil2z
 TvbrokZQca2808v2rYO3TtvtRMtmW/M/yyR233G/JSNos4lODkCwd16GKjERYj+sJsW4/hoZ
 RQiJQBxjnYr+p26JEvghLE1BMnTK24i88Oo8v+AngR6JBxwH7wFuEIIuLCB9Aagb+TKsf+0c
 HbQaHZj+wSY5FwgKi6psJxvMxpRpLqPsgl+awFPHARktdPtMzSa+kWMhXC4rJahBC5eEjNmP
 i23DaFWm8BE9LNjdG8Yl5hl7Zx0mwtnQas7+z6XymGuhNXCOevXVEqm1E42fptYMNiANmrpA
 OKRF+BHOreakveezlpOz8OtUhsew9b/BsAHXBCEEOuuUg87BTQRSeAENARAAzu/3satWzly6
 +Lqi5dTFS9+hKvFMtdRb/vW4o9CQsMqL2BJGoE4uXvy3cancvcyodzTXCUxbesNP779JqeHy
 s7WkF2mtLVX2lnyXSUBm/ONwasuK7KLz8qusseUssvjJPDdw8mRLAWvjcsYsZ0qgIU6kBbvY
 ckUWkbJj/0kuQCmmulRMcaQRrRYrk7ZdUOjaYmjKR+UJHljxLgeregyiXulRJxCphP5migoy
 ioa1eset8iF9fhb+YWY16X1I3TnucVCiXixzxwn3uwiVGg28n+vdfZ5lackCOj6iK4+lfzld
 z4NfIXK+8/R1wD9yOj1rr3OsjDqOaugoMxgEFOiwhQDiJlRKVaDbfmC1G5N1YfQIn90znEYc
 M7+Sp8Rc5RUgN5yfuwyicifIJQCtiWgjF8ttcIEuKg0TmGb6HQHAtGaBXKyXGQulD1CmBHIW
 zg7bGge5R66hdbq1BiMX5Qdk/o3Sr2OLCrxWhqMdreJFLzboEc0S13BCxVglnPqdv5sd7veb
 0az5LGS6zyVTdTbuPUu4C1ZbstPbuCBwSwe3ERpvpmdIzHtIK4G9iGIR3Seo0oWOzQvkFn8m
 2k6H2/Delz9IcHEefSe5u0GjIA18bZEt7R2k8CMZ84vpyWOchgwXK2DNXAOzq4zwV8W4TiYi
 FiIVXfSj185vCpuE7j0ugp0AEQEAAcLBfAQYAQoAJgIbDBYhBKirxr6Vllfig9QtdXK25u9M
 WD0tBQJffU8wBQkWa0+jAAoJEHK25u9MWD0tv+0P/A47x8r+hekpuF2KvPpGi3M6rFpdPfeO
 RpIGkjQWk5M+oF0YH3vtb0+92J7LKfJwv7GIy2PZO2svVnIeCOvXzEM/7G1n5zmNMYGZkSyf
 x9dnNCjNl10CmuTYud7zsd3cXDku0T+Ow5Dhnk6l4bbJSYzFEbz3B8zMZGrs9EhqNzTLTZ8S
 Mznmtkxcbb3f/o5SW9NhH60mQ23bB3bBbX1wUQAmMjaDQ/Nt5oHWHN0/6wLyF4lStBGCKN9a
 TLp6E3100BuTCUCrQf9F3kB7BC92VHvobqYmvLTCTcbxFS4JNuT+ZyV+xR5JiV+2g2HwhxWW
 uC88BtriqL4atyvtuybQT+56IiiU2gszQ+oxR/1Aq+VZHdUeC6lijFiQblqV6EjenJu+pR9A
 7EElGPPmYdO1WQbBrmuOrFuO6wQrbo0TbUiaxYWyoM9cA7v7eFyaxgwXBSWKbo/bcAAViqLW
 ysaCIZqWxrlhHWWmJMvowVMkB92uPVkxs5IMhSxHS4c2PfZ6D5kvrs3URvIc6zyOrgIaHNzR
 8AF4PXWPAuZu1oaG/XKwzMqN/Y/AoxWrCFZNHE27E1RrMhDgmyzIzWQTffJsVPDMQqDfLBhV
 ic3b8Yec+Kn+ExIF5IuLfHkUgIUs83kDGGbV+wM8NtlGmCXmatyavUwNCXMsuI24HPl7gV2h n7RI
In-Reply-To: <70cb224f-a468-40a0-82c7-735c590bdb43@oracle.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-bounce-key: webpack.hosteurope.de;regressions@leemhuis.info;1730983189;3181119a;
X-HE-SMSGID: 1t91nY-00009W-BT

On 24.10.24 17:49, John Garry wrote:
> Hi workqueue and scheduler maintainers,
> 
> As reported in https://lore.kernel.org/linux-fsdevel/df9db1ce-17d9-49f1-
> ab6d-7ed9a4f1f9c0@oracle.com/T/
> #m506b9edb1340cdddd87c6d14d20222ca8d7e8796, I am experiencing a
> workqueue lockup for v6.12-rcX.

John, what this resolved in between? This and the other thread[1] look
stalled, but I might be missing something. Asking, because I have this
on my list of tracked regressions and wonder if this is something that
better should be solved one way or another before 6.12.

[1]
https://lore.kernel.org/lkml/63d6ceeb-a22f-4dee-bc9d-8687ce4c7355@oracle.com/

Ciao, Thorsten

> At the point it occurs, the system becomes unresponsive and I cannot
> bring it back to life.
> 
> Enabling /proc/sys/kernel/softlockup_all_cpu_backtrace does not give
> anything extra in the way of debug. All I get is something like this:
> 
> Message from syslogd@jgarry-atomic-write-exp-e4-8-instance-20231214-1221
> at Oct 24 15:34:02 ...
>  kernel:watchdog: BUG: soft lockup - CPU#29 stuck for 22s! [mysqld:14352]
> 
> Message from syslogd@jgarry-atomic-write-exp-e4-8-instance-20231214-1221
> at Oct 24 15:34:02 ...
>  kernel:BUG: workqueue lockup - pool cpus=4 node=0 flags=0x0 nice=0
> stuck for 30s!
> 
> Message from syslogd@jgarry-atomic-write-exp-e4-8-instance-20231214-1221
> at Oct 24 15:34:02 ...
>  kernel:BUG: workqueue lockup - pool cpus=31 node=0 flags=0x0 nice=0
> stuck for 49s!
> ^C
> 
> Can you advise on a robust method to get some debug from this system?
> 
> Maybe this is a scheduler issue, as Dave mentioned in that same thread.
> 
> Thanks,
> John
> 

#regzbot poke

From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from mail-pg1-f182.google.com (mail-pg1-f182.google.com [209.85.215.182])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4747C19580F
	for <linux-kernel@vger.kernel.org>; Thu,  7 Nov 2024 22:57:42 +0000 (UTC)
Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.182
ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116;
	t=1731020263; cv=none; b=Lx6Lq1S+WNKdyTmKDUcAnqi8mEsYwPRjNbiMPOJ5nVWI8kZVHAtWiACFw+eIitLYXlugtUH4IcjcQhsYCrt6tH1QPV4fqwwLWjS5y67gFaUj3vp8mYX+mjps22Vr8IfDAPozTtC0uXo+qvUCH3olzbIV4n2O4KMnXVryI6avXCw=
ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org;
	s=arc-20240116; t=1731020263; c=relaxed/simple;
	bh=m52yUTLXSN7mqJyraOt9PomUqCPWj+ee9kNEquEdAmI=;
	h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version:
	 Content-Type:Content-Disposition:In-Reply-To; b=JFZcUeXgrvuTzq/MFxByGXBeaGP4vhVWYMtZ/c/JpMk+AYbDMNIMUT5LlrnGIh8dWa7xyy7y9tCq1RPxwhAkLy2REvXeew1YoiKQSiiv1HAGv9UuTF+3tSkBXJ7YmEHdDLzsbQEx0iwB7NztssI98htrONQgD9t1DWFT3mLAPms=
ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=fromorbit.com; spf=pass smtp.mailfrom=fromorbit.com; dkim=pass (2048-bit key) header.d=fromorbit-com.20230601.gappssmtp.com header.i=@fromorbit-com.20230601.gappssmtp.com header.b=2uMyFGos; arc=none smtp.client-ip=209.85.215.182
Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=fromorbit.com
Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fromorbit.com
Authentication-Results: smtp.subspace.kernel.org;
	dkim=pass (2048-bit key) header.d=fromorbit-com.20230601.gappssmtp.com header.i=@fromorbit-com.20230601.gappssmtp.com header.b="2uMyFGos"
Received: by mail-pg1-f182.google.com with SMTP id 41be03b00d2f7-7ea9739647bso1173475a12.0
        for <linux-kernel@vger.kernel.org>; Thu, 07 Nov 2024 14:57:42 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=fromorbit-com.20230601.gappssmtp.com; s=20230601; t=1731020261; x=1731625061; darn=vger.kernel.org;
        h=in-reply-to:content-disposition:mime-version:references:message-id
         :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to;
        bh=26o/4cvY0wiI0+1drLX89Lk/xaFY8ZXJumNp6aN23Gc=;
        b=2uMyFGoskrePX1SMdiklRh7yeHqNdPiwjeQC4j/k1hv3ySknsFfj3YiYOkaOMDA4Ud
         tkTNp6R/eWYlTmn1pGttsG4SdVUZONj0hToe0MibNPeuygX4lr+ZUx9bRldi2q/I3ElP
         Ky3pXOVMmxsu1miU/jOQ2VmfXCE5/IseH5/pWF9Sgiym7H0vXraBCarYoE4k8hL10KlB
         aISIJjvDxwO9I6vtVGXGCGkKz/SwAQN4nN70Js+i1NImyz2+T51e+J/J0AiXhh9OYz7s
         XmTMdAN6Hdu3Xa2SZaMAOd0XEG6kXhKEnd3U8JvoLxJ4YLN+eK7bek6yET/4dvXr0KUy
         8MHA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1731020261; x=1731625061;
        h=in-reply-to:content-disposition:mime-version:references:message-id
         :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date
         :message-id:reply-to;
        bh=26o/4cvY0wiI0+1drLX89Lk/xaFY8ZXJumNp6aN23Gc=;
        b=Rh4S7th9Pol47p9gjkgCjx2rfPXCwBar2/zOIOqxpoCSbnVgInPl/SLAYEPj/uphyQ
         CX3Fg6ZTcrkSvRPIasyNfYlbtJL7yIrJve3lewh54TrmIjhfRXMBYxIjYMkZqPXhJHlT
         ILDRyfn38pl4BULw62cnTDMVZK3qVp/XPDtgF9g080bvE5XrhLtrVPYvWZwhpTnQEvJy
         KTMfpgPEctY9fFBP8R1a8RyL0nPxvE7j9ua0w37m4ZGXFHT11M4QXtmsFGoNFc1gw6cJ
         L1cQQo7rFeIFBUNaMgktbeNAgtJQ6C9qMLYCDUy8LomJJXURkR1cttrdjZOvIUV/hhf7
         h1PA==
X-Forwarded-Encrypted: i=1; AJvYcCUn6jQgSvzZq7yir/lEK9DIm6ETRvVL1TSe4YBlglKA6AKYgT3QfCfjxbNEXkBpMvKcCZW5pD5E+EvII6E=@vger.kernel.org
X-Gm-Message-State: AOJu0YyfLT9Q5FOU1ZUk8pCfHslO2azI9v+yZgauE3PeFlVcGVh4oMD/
	P4lxPnDtGITpQB7cMkRNv4TIe3x0POzITgWKIMjCxuYA1jF58fje5fmIlQ4Y92E=
X-Google-Smtp-Source: AGHT+IH/MI8RKGKO9GoLqS7Sf4/ssMjk1wFFHxWcECDqENvF5I+waceGuptAQk/16n0V94A6yBPXZA==
X-Received: by 2002:a17:90a:d405:b0:2e2:d3a0:4716 with SMTP id 98e67ed59e1d1-2e9b173c50fmr1446878a91.18.1731020261648;
        Thu, 07 Nov 2024 14:57:41 -0800 (PST)
Received: from dread.disaster.area (pa49-186-86-168.pa.vic.optusnet.com.au. [49.186.86.168])
        by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e99a541d06sm4207468a91.14.2024.11.07.14.57.41
        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
        Thu, 07 Nov 2024 14:57:41 -0800 (PST)
Received: from dave by dread.disaster.area with local (Exim 4.96)
	(envelope-from <david@fromorbit.com>)
	id 1t9BRa-00Bagg-0G;
	Fri, 08 Nov 2024 09:57:38 +1100
Date: Fri, 8 Nov 2024 09:57:38 +1100
From: Dave Chinner <david@fromorbit.com>
To: Thorsten Leemhuis <regressions@leemhuis.info>
Cc: John Garry <john.g.garry@oracle.com>, tj@kernel.org,
	jiangshanlai@gmail.com, mingo@redhat.com, peterz@infradead.org,
	juri.lelli@redhat.com, jack@suse.cz, akpm@linux-foundation.org,
	linux-kernel@vger.kernel.org
Subject: Re: workqueue lockup debug
Message-ID: <Zy1F4n9KoNybBmoD@dread.disaster.area>
References: <70cb224f-a468-40a0-82c7-735c590bdb43@oracle.com>
 <4e58d34a-ce45-437a-95a2-3ba21f35bbb5@leemhuis.info>
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <4e58d34a-ce45-437a-95a2-3ba21f35bbb5@leemhuis.info>

On Thu, Nov 07, 2024 at 01:39:39PM +0100, Thorsten Leemhuis wrote:
> On 24.10.24 17:49, John Garry wrote:
> > Hi workqueue and scheduler maintainers,
> > 
> > As reported in https://lore.kernel.org/linux-fsdevel/df9db1ce-17d9-49f1-
> > ab6d-7ed9a4f1f9c0@oracle.com/T/
> > #m506b9edb1340cdddd87c6d14d20222ca8d7e8796, I am experiencing a
> > workqueue lockup for v6.12-rcX.
> 
> John, what this resolved in between? This and the other thread[1] look
> stalled, but I might be missing something. Asking, because I have this
> on my list of tracked regressions and wonder if this is something that
> better should be solved one way or another before 6.12.
> 
> [1]
> https://lore.kernel.org/lkml/63d6ceeb-a22f-4dee-bc9d-8687ce4c7355@oracle.com/

I'm still seeing the scheduler bug in -rc6.

-Dave.
-- 
Dave Chinner
david@fromorbit.com

From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id C83AD47F69
	for <linux-kernel@vger.kernel.org>; Fri,  8 Nov 2024 08:54:28 +0000 (UTC)
Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199
ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116;
	t=1731056070; cv=none; b=H1GN7NP8HmXTZfCOVxnIK0fgFNFoeuwMxXcoQNs4J7edlm7qP/G8XAs3wtPm6VldVyIaAwp8K3zCU9r9JnWdgQ1Qi02HYDwrKchxIJRDCkTznECk5Rb1niGUSnGf+m9OuAlDoPqq8K2TCu9gWaibMLP7hPT2n/D+7/Q4Xnw3yZE=
ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org;
	s=arc-20240116; t=1731056070; c=relaxed/simple;
	bh=VjgEl5vpmDA3y62ssYmWmMIvrNTkWWqVk4IascRENPQ=;
	h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version:
	 Content-Type:Content-Disposition:In-Reply-To; b=DaIwXgkI8jTckQLhwmIfO5DQyG1Du+KomyPgdca7UCRXGe0ma8jfmLQb4xac1vwpJ6bg022uHFSaITe/g7cFt+nPjjtd8fyRFIJwK0ybgYp3Du3v/uPJTXhwo3DtQ8F0ZUrcQksCp4NANhNLWyVyOQyq4fbtLk8oVfLe81zQO8s=
ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org; spf=none smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=fNyUQPoR; arc=none smtp.client-ip=90.155.92.199
Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org
Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org
Authentication-Results: smtp.subspace.kernel.org;
	dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="fNyUQPoR"
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
	d=infradead.org; s=desiato.20200630; h=In-Reply-To:Content-Type:MIME-Version:
	References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To:
	Content-Transfer-Encoding:Content-ID:Content-Description;
	bh=Yf79FXHfTGJLW0g+hLI9OmeHJl+edRD1p43y6vF5eX8=; b=fNyUQPoRE67PhTOM5dhcVHZTaU
	sLDp1NstM8NGXLTSj9HO1hcqsu4rqGCdFJXTmFfTXErdtRCj0eFAKL/l1vasW3JWu4aAtwKwSBPec
	YmVPyM7u0W4GMdoWFjv9AUSf8BJTe0z5sD15eClhTnaLmXqgTWMWfHxOYmCifYUlx143Tn7y0/RH6
	1Q4wUUxcOyETTUFyuI+W4V9kxDQMg5DvroytgBS/4Avi5oqEzAFaJNyFaE0iIRpo12m2tPtLP6I06
	gqzn4sh2ozSpPB21UEaDABtbnQTksqNBSt/jGR32KDtsGVcmNhzo9l11MSjOo1tsolsMYMsS0YcIM
	bk9nkxcg==;
Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net)
	by desiato.infradead.org with esmtpsa (Exim 4.98 #2 (Red Hat Linux))
	id 1t9Kl2-0000000CH3e-07gY;
	Fri, 08 Nov 2024 08:54:20 +0000
Received: by noisy.programming.kicks-ass.net (Postfix, from userid 1000)
	id 4E13D30049D; Fri,  8 Nov 2024 09:54:19 +0100 (CET)
Date: Fri, 8 Nov 2024 09:54:19 +0100
From: Peter Zijlstra <peterz@infradead.org>
To: Dave Chinner <david@fromorbit.com>
Cc: Thorsten Leemhuis <regressions@leemhuis.info>,
	John Garry <john.g.garry@oracle.com>, tj@kernel.org,
	jiangshanlai@gmail.com, mingo@redhat.com, juri.lelli@redhat.com,
	jack@suse.cz, akpm@linux-foundation.org,
	linux-kernel@vger.kernel.org
Subject: Re: workqueue lockup debug
Message-ID: <20241108085419.GA6497@noisy.programming.kicks-ass.net>
References: <70cb224f-a468-40a0-82c7-735c590bdb43@oracle.com>
 <4e58d34a-ce45-437a-95a2-3ba21f35bbb5@leemhuis.info>
 <Zy1F4n9KoNybBmoD@dread.disaster.area>
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <Zy1F4n9KoNybBmoD@dread.disaster.area>

On Fri, Nov 08, 2024 at 09:57:38AM +1100, Dave Chinner wrote:
> On Thu, Nov 07, 2024 at 01:39:39PM +0100, Thorsten Leemhuis wrote:
> > On 24.10.24 17:49, John Garry wrote:
> > > Hi workqueue and scheduler maintainers,
> > > 
> > > As reported in https://lore.kernel.org/linux-fsdevel/df9db1ce-17d9-49f1-
> > > ab6d-7ed9a4f1f9c0@oracle.com/T/
> > > #m506b9edb1340cdddd87c6d14d20222ca8d7e8796, I am experiencing a
> > > workqueue lockup for v6.12-rcX.
> > 
> > John, what this resolved in between? This and the other thread[1] look
> > stalled, but I might be missing something. Asking, because I have this
> > on my list of tracked regressions and wonder if this is something that
> > better should be solved one way or another before 6.12.
> > 
> > [1]
> > https://lore.kernel.org/lkml/63d6ceeb-a22f-4dee-bc9d-8687ce4c7355@oracle.com/
> 
> I'm still seeing the scheduler bug in -rc6.

But that WARN you reported earlier isn't there anymore. So what exactly
are you seeing now?

From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id B6884193092
	for <linux-kernel@vger.kernel.org>; Mon, 11 Nov 2024 13:22:34 +0000 (UTC)
Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=205.220.165.32
ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116;
	t=1731331356; cv=fail; b=T/oyakVV3aUK8Xf1sYs8XkVa9FQ83d7NqqmZvbui+w4aAjZCRBXkI1Jgoyqx5W8/lq21AdQCxoWHvIF13nl1F6JOwZ4EwUxNMLX213gpvpZvTj1slY/dbiNC1R/uyRan2WgNNsA9ao+m0gmhVlI8g2u9UYKOsWnAVkAySCdljpk=
ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org;
	s=arc-20240116; t=1731331356; c=relaxed/simple;
	bh=wvHLahzYm4C0A8fPKC83Szq5WlL0rMQHNOiqdiZtXuM=;
	h=Message-ID:Date:From:Subject:To:Cc:References:In-Reply-To:
	 Content-Type:MIME-Version; b=MRd3P2zt5KKFCOSLARKOzFcr60ThvLf9ru9H5zHBkwaTfOqn85rAwhX6YnuR+vY1aSJ+6aJ6oPKx8oRfdU2sVRtMo0DgkGYWUNeb0BP1WOGYgY3awlc1TgBLiyuDZiaU7OcucOIQb8Qp416hHTLFn4OgYz7jJE9F5e0lUAFxA64=
ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com; spf=pass smtp.mailfrom=oracle.com; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b=ELHlIIGT; dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b=aBWSYdD8; arc=fail smtp.client-ip=205.220.165.32
Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=oracle.com
Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=oracle.com
Authentication-Results: smtp.subspace.kernel.org;
	dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="ELHlIIGT";
	dkim=pass (1024-bit key) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.b="aBWSYdD8"
Received: from pps.filterd (m0246627.ppops.net [127.0.0.1])
	by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 4AB9spJc016893;
	Mon, 11 Nov 2024 13:22:16 GMT
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc
	:content-transfer-encoding:content-type:date:from:in-reply-to
	:message-id:mime-version:references:subject:to; s=
	corp-2023-11-20; bh=EhNs6FFRWl7wVWzvuQ7O+miJJ12x2CA8CmyTP3EAnBI=; b=
	ELHlIIGTh4axRaTyQfOkFt0gi6K+mcfRcC2i+QqZm0+9NylzC71QOcTYhqEaIpJo
	A5OcodTawCNHIjlmk8yeyxMNsrcLqusGKa2E0lHOKbG6Gf3PxY4+r7djbTzvcacu
	7F3zikMpI9z0on/cMZJ421H3wA+YqtV1MKdYgwpZAme/7ca4ZHW8YRN0iqBXijqF
	5lf9TQAHda9DuWBKNI2HfdOEKhC4lNp1krG8lnXpGjlY/BBTwEDsYbbdt3ClsWIq
	R/BC4NvUSc3SbWm7tslbDa2o4WE/gLYgqxhiNsDUGxbQh7U04YO0aki7K6vGKK1p
	LdjYUmEbljvV/rIqYKHcBQ==
Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129])
	by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 42t0hnacnc-1
	(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);
	Mon, 11 Nov 2024 13:22:16 +0000 (GMT)
Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1])
	by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 4ABCafHE037193;
	Mon, 11 Nov 2024 13:22:15 GMT
Received: from nam02-dm3-obe.outbound.protection.outlook.com (mail-dm3nam02lp2047.outbound.protection.outlook.com [104.47.56.47])
	by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 42sx671buk-1
	(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);
	Mon, 11 Nov 2024 13:22:15 +0000
ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none;
 b=u1hIibd5rzSjiMjgaF9IDv5eBL8PjkQI2UyiBLa1vt2Sjt+PZAJkc73uaoVgvbSzj3VYM6yo1n6grAGj9fTwUhfF9skQQq3Rb+i9hGau6Z2KuH0DA4W4k0Z4NZZcMy//wX3dhyE179UmbMQ5seMOIh2phSjItNNS+Ksl6tYSpOUmG+ErqK7daZNm3zBei5h9CoA8Ug1/1JHeq6M9uWhfGZDJFQ3LiGMxyUF10lVR61LZbbA3BSo2VEtC2RLUm2nBylpWGQCJyZnre1a+eWip06G8IQpGpw+JoQr8EgaMvhhqM04Xo3aKqh53Qhk+Ru6patWUWZmDGqDxvt8C5MeHPw==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector10001;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=EhNs6FFRWl7wVWzvuQ7O+miJJ12x2CA8CmyTP3EAnBI=;
 b=HutC4ZbqSdXlsVJ2VgTLOob9hA5+okZfqsqfUAQNqTMQb+vrjoJCZMnwQUa2y13MTEc5ldIaykb8tGPu3iIAPX0WTqc9WGudzk/8hzY9UHRg/N4TIwbArl4x+Bw9NTWjWygZDfmVHGxylPb3HpR910U4Vgrq/WdBDn3FEXc+4HGh/sPItiMUrNEEexohTuQJ+aCmJuWv+N84FV6FC+i7XxfNYs6WHZe2AelkUZwTaElReO/f4M03lhZ9X9043G60NJfpga3rLOFIuRpwaxSowiL83v0OZd0tJop2vpTFiifAWCSGcYNR8hoVCrEdh0npEWLD+UjiSvdQhw/oS6vMHg==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com;
 dkim=pass header.d=oracle.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=EhNs6FFRWl7wVWzvuQ7O+miJJ12x2CA8CmyTP3EAnBI=;
 b=aBWSYdD8eC4wLIM+KR6Vl64W+XGgLd4/Hh1ALXqq6LM6tdZjn1ukme4vzzvz2cLOai4sNqnz/IB6kMFH7BMV5lZv8EEv9AuqYdVbZZ6vDGRt96tq7dLffxp4ERrx3koeOTBFvGzL7kI3JfIET1jqaTnCsmLTAgSYQD99KcE4jls=
Received: from CH2PR10MB4312.namprd10.prod.outlook.com (2603:10b6:610:7b::9)
 by DS7PR10MB5023.namprd10.prod.outlook.com (2603:10b6:5:38d::16) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8137.28; Mon, 11 Nov
 2024 13:22:13 +0000
Received: from CH2PR10MB4312.namprd10.prod.outlook.com
 ([fe80::fd5e:682a:f1ac:d0a2]) by CH2PR10MB4312.namprd10.prod.outlook.com
 ([fe80::fd5e:682a:f1ac:d0a2%6]) with mapi id 15.20.8137.027; Mon, 11 Nov 2024
 13:22:12 +0000
Message-ID: <3a9bd52c-b0b7-4c6d-a7d7-2198289fab27@oracle.com>
Date: Mon, 11 Nov 2024 13:22:02 +0000
User-Agent: Mozilla Thunderbird
From: John Garry <john.g.garry@oracle.com>
Subject: Re: workqueue lockup debug
To: Peter Zijlstra <peterz@infradead.org>, Dave Chinner <david@fromorbit.com>
Cc: Thorsten Leemhuis <regressions@leemhuis.info>, tj@kernel.org,
        jiangshanlai@gmail.com, mingo@redhat.com, juri.lelli@redhat.com,
        jack@suse.cz, akpm@linux-foundation.org, linux-kernel@vger.kernel.org
References: <70cb224f-a468-40a0-82c7-735c590bdb43@oracle.com>
 <4e58d34a-ce45-437a-95a2-3ba21f35bbb5@leemhuis.info>
 <Zy1F4n9KoNybBmoD@dread.disaster.area>
 <20241108085419.GA6497@noisy.programming.kicks-ass.net>
Content-Language: en-US
Organization: Oracle Corporation
In-Reply-To: <20241108085419.GA6497@noisy.programming.kicks-ass.net>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-ClientProxiedBy: SI1PR02CA0015.apcprd02.prod.outlook.com
 (2603:1096:4:1f7::18) To CH2PR10MB4312.namprd10.prod.outlook.com
 (2603:10b6:610:7b::9)
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-TrafficTypeDiagnostic: CH2PR10MB4312:EE_|DS7PR10MB5023:EE_
X-MS-Office365-Filtering-Correlation-Id: a1db8d68-c830-4702-cafa-08dd0253da35
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|366016;
X-Microsoft-Antispam-Message-Info:
	=?utf-8?B?UGpGb2Y0Z3RtejNXN1E3dlY4MEF2b1I1T2cvdG9NeWljQ3JJa2NwaGl6MFZB?=
 =?utf-8?B?K25DbUNGWHFibFdOM1RCRjNmWjhoa3JoY29GWlQyQURRR1ZHNXRIRDR1bGIz?=
 =?utf-8?B?KzkwWXIwWUFsRTVKWmpsZEhFZHFQT3J3bE5xMnVOL1c1VC85Nzl5dHFwcDdN?=
 =?utf-8?B?TU5zK2s4V3Bjam1icUVMTUJEbnBreTJqQXYwN3hVMUd0SVREUmpxY3RpTnli?=
 =?utf-8?B?ejJTckUyU3l1VkN1enpyQ0ZQNUU3enBFckFFcFh1VjluT2d1TWZhNUs4VWJs?=
 =?utf-8?B?dDJSR2Y3azVQZ0ZXS3UrVVQ2L0poMDI3WDgvOGlZTXU2Zm03NnhYOS9wWCsr?=
 =?utf-8?B?QVRaRGgySDA5T3N6cUZiRWpWZG4ycmVJUjI5dUNRUFNpdUV5OHJjbjlTUnZB?=
 =?utf-8?B?Uk1lL09rYmkvVHJObUxPZ0kvMUUzcitsUC90UHlzUmROU0lWSDgvTXl4c3l2?=
 =?utf-8?B?VjNBaWhDMHhmcnV0TGIzVUtqbWNXbWVpMXI0SVFlOWdYTDcwRk5HQTVBNC9a?=
 =?utf-8?B?WVBkM0tldzZMVkpCVkQ1M0ZQNTBTQWhCZy9RdjRwTldaYVZIditWTGhBT2lv?=
 =?utf-8?B?M28wRDBrcWZITk1tRjdkN3RtN0kzdlJuWncwM1lsSk1yeWp4NDB2S25rN2FU?=
 =?utf-8?B?WXk4ckx6SUV6a2w1N2puWTU2ekJnOEpHcVN4Zk0wRC90Uk9aZkw3VmFEdjNW?=
 =?utf-8?B?KzlZdDNHYUY5VVkvd0FnOE4xRW9PRHpGWUJUTE5VNHF0NWdyUVhhZXVVM20v?=
 =?utf-8?B?QnpoMm5tbjlreFNpQjVhUzQxWVAyMnlqYnNmYnJSOW9uV0tXVGVyL3lyWDZY?=
 =?utf-8?B?bG0xN28zb3l3STNkMTJKR1N5ckhCSVhpR1M2aTRlR2hlbThIVG9zVUNNL0RK?=
 =?utf-8?B?MVRtU0hUN3crZzlCeHkzaVExaXZ1SENUNVRnMzE2QlFtTU0vcjBpL0FIV1R6?=
 =?utf-8?B?Y05KaGFYNHNYZ29rS3BubDB6Y2w2eEYwZnpYQzdrcU5yV25DYnpqYng5Vzlt?=
 =?utf-8?B?YS9DaXBsa0ZIaVlxelpaWFFxWjlHd0YxbElGd2F1cUpGOVcvZzBwTzVnUnJ3?=
 =?utf-8?B?ZEhBdGJSMnFaUmJSL2cxcTl2OEgyaU1VeGpPSGtDdjkyS0lxM1ROUWFiMVVp?=
 =?utf-8?B?OFdjRXFUUXFFTmtFYlp4VFZLamo0VnVxRTdxVTdxNVQ2aDBmUE5IWkxHKzBV?=
 =?utf-8?B?RnlPSDN1bmVjL1JYWE9zbGd2U0VvQkRaclVYRjY5c1Baek1WTnJkZTFKdUtm?=
 =?utf-8?B?YnZLcDhpN2JrbXRMdmtIMXBpV1VsSWlwbnN3WHlpVXNic2ZmMjVrSGdPVTdv?=
 =?utf-8?B?M2pFbU1YMlA5YXV5SXFhelFndHJCN2dDTE1PU1NnZ1lTMk5XY2ZRUFVFMi9F?=
 =?utf-8?B?eTV5L2gyVVpRbi80R2x1cGxpWFNJZmRRYkhPemlwVnF0VUlvenkrZlppU2h1?=
 =?utf-8?B?TzQ1RGpVOEFKOTlwM1hkNkxuR283ZjZ1NEhEeEpwVWZ1SUFQOXh4VGZVVklv?=
 =?utf-8?B?Qm52eGlZbnJuZ0VSWXpaWWo2YngrKzlBaU5yWldlZyt2eWdGdmhHMmNETXhN?=
 =?utf-8?B?UEgyekI3RW9mN1VrK1o3aWRZTEhaSCt6SUMvNXBPNXNGM1ZUdEdOZy9UTFAr?=
 =?utf-8?B?Q2t2SHhJRHZXc1U2YU5WWkFOcEY0K3l2aElVOXdUYUwrd2ZZOGQ4Z3J0QkNL?=
 =?utf-8?B?aDFQUzl0cHhtbVFyQnEzUjJFVUJMWHE1NTBNbTR5WlZKcUxQeGdDc21pNVJx?=
 =?utf-8?Q?tMSFlN0devtLNp7aeI021v2JeAwPw4RK3GADqgE?=
X-Forefront-Antispam-Report:
	CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH2PR10MB4312.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(366016);DIR:OUT;SFP:1101;
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?d1dGV2lvN1hjRlEyWGsxRUY2NEdsM2Q5QTBoWnhnR0x6QjZQSzhheHJkV0xD?=
 =?utf-8?B?V3F6dFJnYlFmMjZSbXYySW00eGR2cDdGYVhVZHlVK1NBSmdqNWVrdjEwRGMv?=
 =?utf-8?B?ZXFRc0wwVVVyRzQwb05yczJHWWxvZnRSRWdTNEozS0VoZUhvR0wvSUprRzhj?=
 =?utf-8?B?cnRpREFvNVBiZ01TRnF6TFFlalUraWg5NHdBK01DbVlOcGFJWEhMZWRZSU9l?=
 =?utf-8?B?VjRhOHFtNnZ4dUtRbGtuNVJJc0NPcEJFbStUU1RqREk2UjgycmpiczBIcTl0?=
 =?utf-8?B?eWpsajIvYUhRV2NjazA5cEF2Qm92Qlh1MWFSZ0E1VXZEa0x1VWhndUlUeTJX?=
 =?utf-8?B?c05oazJxQ2tkdkQzZzBrdHk4bUlTb0t0aDJKMHpwWUF6WXBiOTQwTVkxeTVp?=
 =?utf-8?B?cEcySUkycW1GaEVDRmpQOGwvaldvR0FhOFlGNTZsZEQrZnlmbys5aUFQekpZ?=
 =?utf-8?B?dkcwRHdyK1NIQjFUblZKQ3RrakZkbVlhUlNjWlNrOCtOL1B4ZlJXT3NLamRz?=
 =?utf-8?B?a0FWUmhHSWthRXhHZkhvdGhsSTIyME5Sd2NwRGN3UUpsSGFnVkFROXk2OG5n?=
 =?utf-8?B?WS9haFphS0M1OHBJUysxNEdTWmozMkxvNkNaWE9yekYwUkdtSFhyTDRpejh3?=
 =?utf-8?B?UkI1Q3U3QU9PQXo2c2h1QkJjeDdTZ2doUzRHaU5seERYYmphR2l0YnZxOXcy?=
 =?utf-8?B?UmpIYTI5Ym9zTnVIZlNsMXpCZitUUDVxNVNyOUt3WitldGM4QVdtSTVsUm8r?=
 =?utf-8?B?TjBiMG9pRXRFdE9kazJ5dVpoTjltVWlGTis0cStublVqM1B6ZVhPNE1kVGNI?=
 =?utf-8?B?Vis3RmZ2RnN3QVFKczZXeEJPVGI4S1NIRGhseFJPYXFQR0tINWdZRm45c0pF?=
 =?utf-8?B?a2JIWk1ONXZuUlJtWk96NmJoa1VSa3VjT2d4blovalBvalZnM1ZJN2xOQlNy?=
 =?utf-8?B?NTNLRVhqcUpaNnF4RS80U0VZVndwUm03NUxqR3dSVmZtMG1LNEpaU3hZSS9E?=
 =?utf-8?B?blVJVWtJdnBucW1WRWxETmptMGE3S2toSGJEbEc1K2k5S0tXYnVSQ1FaM1B4?=
 =?utf-8?B?b1VhSmdjRFdNYzdSQ09TTWxMRTdzc0tTSGJoUDRUWGo5U0M4c1RGakxlcElv?=
 =?utf-8?B?NXdiN2JXU2ZSLzZqdzFmSWJXTEQ3bVJ1dWpZTFVSZUczV0s5SHhENzBFeFVO?=
 =?utf-8?B?UGthSlowVGd2L05ScjdJN2szOUdiN1BMNElWL3ZRR3o1MG8rN3pGM3pvdkRk?=
 =?utf-8?B?U1N3NXhZd2h4cndRTk1TOGFNcHdOOTc4WEtsNXNudmVocEptMEl5QzZhODh6?=
 =?utf-8?B?UWd6WUtVUlhTdGJKbEhSTEVrSFVCbzJoZXlmRFJLeCtSbkxGTTMxNnF4ZHRq?=
 =?utf-8?B?WGRDdVlqZnFYQXZEdk9rRXFacDR0OC90QUxMTFNPS0Z3Y2xIbkZhdVZPSFdD?=
 =?utf-8?B?L1pDNDF3NE5tbHdNMjA0MmJ6MGhJTlpMYlE2YWdmRzdDdi9lRkVKUDNoSldt?=
 =?utf-8?B?aG9zTFUvN1BEUkJzNXRVWkZxZ0hoVWw2VDBPRFVDdDRlc2loY2p5SHBOUSs2?=
 =?utf-8?B?TVFRYUN1a21kMkpBUjlMUFc1ZEQxSmNEcGFSM0VldVRHWVY2SGZLL2JVcVNm?=
 =?utf-8?B?THBudmpkcXdTTmZQeWhlekJVSU0xTzgxS0c2NC9LUFBKR3p4RmpoQ3V1SkpQ?=
 =?utf-8?B?Um5XaHVleEtyRTh6MDNuMVk0UVl5cjhudzdHakYvdVlheEdsTnZrV2hoeFpi?=
 =?utf-8?B?ZzJLN3RLeCsvc3NNNExqYmFjaHB1UkVjRzlFZTRDZjJVaWdoMElzTDdnREZs?=
 =?utf-8?B?QmYwQVlGZWRzd3E3Zjl2NHFJVnpvdFBLRHpXL3pWeXVMay9EVFd5YTE4NGZH?=
 =?utf-8?B?WXVzdTk0T1NwZGF3OVY3cjRCUjVqcW53M3FqY3cwL0dKMEo1ZThYeFJFb3B2?=
 =?utf-8?B?bHo1bFhvZHVvME1mamFndHpkVCs3alZ1TnJPWXZFeXYvTTVxd1U0MHZOUGZN?=
 =?utf-8?B?R0x4ZWNwYUJDbnlpckd4OWk0L3BaWnJrU3JPRm40bEk5bXRlUEdvMUUyOVda?=
 =?utf-8?B?cUp2aHRId1JYc3RRZ2VvcFpEOXBCanA0b3RzNTY1NnMwVTdleExZRVhITTFK?=
 =?utf-8?B?dWZzT0JPL3ViTzlDZ2ppNThadFd0SGhxSlc5RHQzd3NKYzZiTWY2aGZGUTNS?=
 =?utf-8?B?V1E9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0:
	 
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: a1db8d68-c830-4702-cafa-08dd0253da35
X-MS-Exchange-CrossTenant-AuthSource: CH2PR10MB4312.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Nov 2024 13:22:12.7111
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: HLN2vydEVo5RLIFKzzGlUWDbrJMbTFfz0NX90bxhSahddLOgBRjrOHaU5HAfRWwQ3kVSCQBAXCunIwI9oDiHtw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR10MB5023
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.62.30
 definitions=2024-11-11_08,2024-11-08_01,2024-09-30_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 spamscore=0
 phishscore=0 bulkscore=0 malwarescore=0 suspectscore=0 mlxscore=0
 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2409260000 definitions=main-2411110111
X-Proofpoint-GUID: YfFoD4GNzU4why6sdRGmhaosGSjr8_Ox
X-Proofpoint-ORIG-GUID: YfFoD4GNzU4why6sdRGmhaosGSjr8_Ox

On 08/11/2024 08:54, Peter Zijlstra wrote:
> On Fri, Nov 08, 2024 at 09:57:38AM +1100, Dave Chinner wrote:
>> On Thu, Nov 07, 2024 at 01:39:39PM +0100, Thorsten Leemhuis wrote:
>>> On 24.10.24 17:49, John Garry wrote:
>>>> Hi workqueue and scheduler maintainers,
>>>>
>>>> As reported in https://urldefense.com/v3/__https://lore.kernel.org/linux-fsdevel/df9db1ce-17d9-49f1-__;!!ACWV5N9M2RV99hQ!K-drGDW_XBDuFwUrnEBHKIab7gT2eumqlwIEviGOHbLBedwmMvP_yJGM98ikNAU5uTtes3Ig7Lk40CG3652p$
>>>> ab6d-7ed9a4f1f9c0@oracle.com/T/
>>>> #m506b9edb1340cdddd87c6d14d20222ca8d7e8796, I am experiencing a
>>>> workqueue lockup for v6.12-rcX.
>>>
>>> John, what this resolved in between? This and the other thread[1] look
>>> stalled, but I might be missing something. Asking, because I have this
>>> on my list of tracked regressions and wonder if this is something that
>>> better should be solved one way or another before 6.12.
>>>
>>> [1]
>>> https://urldefense.com/v3/__https://lore.kernel.org/lkml/63d6ceeb-a22f-4dee-bc9d-8687ce4c7355@oracle.com/__;!!ACWV5N9M2RV99hQ!K-drGDW_XBDuFwUrnEBHKIab7gT2eumqlwIEviGOHbLBedwmMvP_yJGM98ikNAU5uTtes3Ig7Lk40NodyRwu$
>>
>> I'm still seeing the scheduler bug in -rc6.
> 
> But that WARN you reported earlier isn't there anymore. So what exactly
> are you seeing now?

My problem reported in 
https://lore.kernel.org/lkml/63d6ceeb-a22f-4dee-bc9d-8687ce4c7355@oracle.com/ 
seems to be fixed after rc5. rc6 and rc7 look ok. But I will test more.

Any idea what could be that fix?






From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8A63C1BC07B
	for <linux-kernel@vger.kernel.org>; Mon, 11 Nov 2024 21:01:48 +0000 (UTC)
Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.180
ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116;
	t=1731358910; cv=none; b=bw3bnGXVBlNTmAbPjRSyt7T/2KAvyXI/NtqOEngp0bHaLo4at/P7tTMLZJQmcHd+hD2ayFjfYzNd4Iimh81x+1du0Tb0Tjixx3fEUcSW/2WMlcITY6cNM5JVkr2to1eJMiZ5Mm6OqKPcoMJTWodF5j79CEk9oVch7qwbfyoRGWU=
ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org;
	s=arc-20240116; t=1731358910; c=relaxed/simple;
	bh=NsMJ6sflvPpN5Om1zPyJEDYX3eOfkFpvEC4GUB5w07I=;
	h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version:
	 Content-Type:Content-Disposition:In-Reply-To; b=XIVLM+MhQrT9TGDiEWhRpBPe5xMdEqBlPtHjvWGiddV5aHdR1hhlOXI7U9YbHePFKwZ24lu27c3/XbF6cEn7b86+i6iAPVhcxPZoKxazPDjcu0mDXrv1wLOJWmPeKMQURlu8R5wC8D1wTKLX+GQyNoNdMb4UuGPv3JjPm3P5k9E=
ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=fromorbit.com; spf=pass smtp.mailfrom=fromorbit.com; dkim=pass (2048-bit key) header.d=fromorbit-com.20230601.gappssmtp.com header.i=@fromorbit-com.20230601.gappssmtp.com header.b=P42VEiY3; arc=none smtp.client-ip=209.85.214.180
Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=fromorbit.com
Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fromorbit.com
Authentication-Results: smtp.subspace.kernel.org;
	dkim=pass (2048-bit key) header.d=fromorbit-com.20230601.gappssmtp.com header.i=@fromorbit-com.20230601.gappssmtp.com header.b="P42VEiY3"
Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-20cdda5cfb6so49408185ad.3
        for <linux-kernel@vger.kernel.org>; Mon, 11 Nov 2024 13:01:48 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=fromorbit-com.20230601.gappssmtp.com; s=20230601; t=1731358908; x=1731963708; darn=vger.kernel.org;
        h=in-reply-to:content-disposition:mime-version:references:message-id
         :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to;
        bh=NEoBXDwxEuaRMLtjNe/ebt/WXN49k6ZubP/KFubYcXs=;
        b=P42VEiY3r3lIwuuapuyGUmVZ/PDkG4JCXKFoifocfRADPWBSWUFl2N5Xvdo9jQCxU9
         G5Yhc8HUOaTALFtjzkvg0E6qNmQ2PUqmzamvF1f516jkCO++Hg7XB8IzVBVJAWQeCKKt
         gGFypMxJFI3pAdGqI0MEZm90k1gwuu8A+xTpnPPEfB9OPHcoGe4gW8Rm73L7FWkkEGuL
         jcT/iBgPKVxjzHxilRHlX3GRPCZnSLoqReegwJTKaB2w+09ZUE+PhF1kDG6Nt2IiYT8k
         ibmwi60q0rrVPo4pHRzXbCJhm5NDe7EhrEnQDUDrG/V1RmPrBV0sSY3yfXdDz5bVwEaR
         Q4+g==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1731358908; x=1731963708;
        h=in-reply-to:content-disposition:mime-version:references:message-id
         :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date
         :message-id:reply-to;
        bh=NEoBXDwxEuaRMLtjNe/ebt/WXN49k6ZubP/KFubYcXs=;
        b=PPb1otCau79SynsnQ1/KMl3PTPT6xtrTWqmeqEH/1ntz/0jmkNcXTKr5Sj7bmssWZa
         Dnah4iWlV8zvy/wIt6KPMx5moz1BTNhySckQqtaTDk2elALQ1a94AdH+3+3CytAXFSBw
         SapcYF20+89x2vWzzfK4EdnDbQVHknnQ9l6/hrXrB5YzZ9nzrPgLLAKMH02SDL4X0yGT
         wSlVJyFel72nFieArLADF8pfek9wi9vx6mtZnVlynqy5/QGiF2kKt0FPqy+LFVvaXxrn
         rLcwfMCljlzmGWS97qD6hS54A8Ts63ZJsBslLD0HLzCA9djmXFkZGowOovJO3kNjO+XB
         a3qA==
X-Forwarded-Encrypted: i=1; AJvYcCWVlQAuK/0LRZz4K4Vz9cb/ojVct5G6N6hKhuwM+46qspcEYywRmF0Z3mTlIeXfrvE1oxJvfcsOudr3XUs=@vger.kernel.org
X-Gm-Message-State: AOJu0Yxzm/jINUvf06rF8FflgCtK7hWlIXdUOx/E41y/btKj1k6YHkc5
	kf1i+WJaKAczsT5AyIPsAvKiNGEgGJ9/9+sqL0E2OTE1cWW/arU7UlAYX8K9CyA=
X-Google-Smtp-Source: AGHT+IG7tW946w3/G7AlvtKGoG4TioAyKsozsSRMpMaW+9Ajdm0/jT0ud0ACSeGosTxtaiT1H0m07g==
X-Received: by 2002:a17:902:e747:b0:20c:5e86:9b53 with SMTP id d9443c01a7336-211834dff33mr198742645ad.3.1731358907875;
        Mon, 11 Nov 2024 13:01:47 -0800 (PST)
Received: from dread.disaster.area (pa49-186-86-168.pa.vic.optusnet.com.au. [49.186.86.168])
        by smtp.gmail.com with ESMTPSA id d9443c01a7336-21177dc9f23sm80547065ad.29.2024.11.11.13.01.47
        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
        Mon, 11 Nov 2024 13:01:47 -0800 (PST)
Received: from dave by dread.disaster.area with local (Exim 4.96)
	(envelope-from <david@fromorbit.com>)
	id 1tAbXc-00DLQI-27;
	Tue, 12 Nov 2024 08:01:44 +1100
Date: Tue, 12 Nov 2024 08:01:44 +1100
From: Dave Chinner <david@fromorbit.com>
To: Peter Zijlstra <peterz@infradead.org>
Cc: Thorsten Leemhuis <regressions@leemhuis.info>,
	John Garry <john.g.garry@oracle.com>, tj@kernel.org,
	jiangshanlai@gmail.com, mingo@redhat.com, juri.lelli@redhat.com,
	jack@suse.cz, akpm@linux-foundation.org,
	linux-kernel@vger.kernel.org
Subject: Re: workqueue lockup debug
Message-ID: <ZzJwuCvLv5ZEIT0e@dread.disaster.area>
References: <70cb224f-a468-40a0-82c7-735c590bdb43@oracle.com>
 <4e58d34a-ce45-437a-95a2-3ba21f35bbb5@leemhuis.info>
 <Zy1F4n9KoNybBmoD@dread.disaster.area>
 <20241108085419.GA6497@noisy.programming.kicks-ass.net>
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20241108085419.GA6497@noisy.programming.kicks-ass.net>

On Fri, Nov 08, 2024 at 09:54:19AM +0100, Peter Zijlstra wrote:
> On Fri, Nov 08, 2024 at 09:57:38AM +1100, Dave Chinner wrote:
> > On Thu, Nov 07, 2024 at 01:39:39PM +0100, Thorsten Leemhuis wrote:
> > > On 24.10.24 17:49, John Garry wrote:
> > > > Hi workqueue and scheduler maintainers,
> > > > 
> > > > As reported in https://lore.kernel.org/linux-fsdevel/df9db1ce-17d9-49f1-
> > > > ab6d-7ed9a4f1f9c0@oracle.com/T/
> > > > #m506b9edb1340cdddd87c6d14d20222ca8d7e8796, I am experiencing a
> > > > workqueue lockup for v6.12-rcX.
> > > 
> > > John, what this resolved in between? This and the other thread[1] look
> > > stalled, but I might be missing something. Asking, because I have this
> > > on my list of tracked regressions and wonder if this is something that
> > > better should be solved one way or another before 6.12.
> > > 
> > > [1]
> > > https://lore.kernel.org/lkml/63d6ceeb-a22f-4dee-bc9d-8687ce4c7355@oracle.com/
> > 
> > I'm still seeing the scheduler bug in -rc6.
> 
> But that WARN you reported earlier isn't there anymore. So what exactly
> are you seeing now?

I just went back and checked the logs. It does appear I was running
an -rc5 base kernel, but that just makes me really confused because
I was running mount path debug trace_printk patches at the time and
that kernel was clearly built on a 6.12-rc6 base.

Yet, apparently, despite getting the tracing output I expected to
see, the kernel still reported as 6.12.0-rc5-dgc+. I suspect some
kind of weird delta rebuild issue (possibly some of my machines have
dropped time sync), but that's my problem to sort out.

IOWs, ignore this, I'll do a full rebuild on -rc7 and test again.
If you don't hear from me again, I haven't hit new problems.

-Dave.
-- 
Dave Chinner
david@fromorbit.com

