From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6ACF4C4332F for ; Fri, 6 May 2022 17:57:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353411AbiEFSB0 (ORCPT ); Fri, 6 May 2022 14:01:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50518 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232138AbiEFSBY (ORCPT ); Fri, 6 May 2022 14:01:24 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0854F6D3AB; Fri, 6 May 2022 10:57:39 -0700 (PDT) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 246FW2al030007; Fri, 6 May 2022 17:56:15 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=message-id : date : subject : to : cc : references : from : in-reply-to : content-type : content-transfer-encoding : mime-version; s=corp-2021-07-09; bh=dvzpIwt5+QcB24+EByvUA0dnLmNi8FJ99aXDZjGDbU8=; b=szHQF+8lv4TfY1qZu8KtHdXOMO9JhGkuR7v1rSdjMV4+gSaybmTsGN8hmPH/hygcgbzM wlUmNy7oBR8CI8OxlJtZvZwNSS9y6VYhUlnmcYdJmirqintgM+cAW8sIWHhmAEzl96ke Qg2gidSPDzMWZvbK/GtLGcaPxV5Cl0JwAVgP/V4FW9TqVBzAcjjJbhvwOKbLJk83q6vH fHALVF7as2aykVD8nao2EcEQPSBvBIlkSz9ZwY1kJPqxZZgRVeWjamfY9iKwC98Pqzrq RhlSyNVV0Lg0X6qXDT7qVr4T5e/IIKXBp/hCX+pCZD+0QlzaunN8EkNa30FYRslqUeUF CQ== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3fruq0pvvs-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 06 May 2022 17:56:14 +0000 Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 246HtSCU026597; Fri, 6 May 2022 17:56:13 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2175.outbound.protection.outlook.com [104.47.59.175]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3fs1a8k75y-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 06 May 2022 17:56:13 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=eGm4n8FtnVnsiGxgrxhhbwHGduLBBRYr+Tw430xh9UjHuFfX7f/CS+BlALtj5KKGculHUPaT4K2xioGdhU64Rw7B9PbmmtA9rDzD6GU7h2ZfHonm2lOWOxHEyE+zSu0r3iqgcye/c/8+eH4QoQU3ahYHy9oImlgCpXl/2Q1sdehLoxWKfihJMU7uvBDfyF3NAVVHrGMhlvgvFCQqu8WD1RRXXRoFn8BwXkBJB9TvxvF+IKk/E8u04F9/AXO0KbRXqHZ5Dw6fzeXpPZlAeXvqOD+NbMznrbOlV8DoX+x8TBC+/JCQxYv4mfknHOkn3fUGGuguAk85tDplEoI+d0mbsA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=dvzpIwt5+QcB24+EByvUA0dnLmNi8FJ99aXDZjGDbU8=; b=gxziFOax49XChajY/51gj+wxvP+YvY7/0L0RX58bwzaMj08zXZMpYcdVRRZ4MvcqljJYJ7W1SkEsWJIMjZdA4LuZRXMX+nK78+FUDV5xkmmX6rjXrWxzN/bEmbUQIuSOgCOqHGBWxQg1f/oDTCBdj3X36Y457r5XTPBSjFLI8NPs8J0UA52FWW5ZEIw+DECk4ClRK38X138Jt/nISd9PNEPdJkih40mSS1gJPEFBxkXhvnBbvlUUNlrCNpmYp5Rq+FVBOX+obQvXNHeB2eXLih2HNo9hmjatda7pQ/pBOHDtfoBeWRI5FzcsmzvVzBV9xArTUeIhMzdKaA8bnCIRHw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dvzpIwt5+QcB24+EByvUA0dnLmNi8FJ99aXDZjGDbU8=; b=ijtx8ilcoPusfnLBiELAw+1Q1dxZpPfgxAjUAl5LUHB4gJoUdMQQ7X8Rm4QPJl56ofP14pS5wJTjFvxLRwnTXGFa+/iV6qD3lKiS4QWEiPRnvevGkKqe6Q7S1iLAWhC92KbO+pvQob/9uvxo7gOXZMYydkc1WyfWflbt/B1CzVM= Received: from DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) by DM6PR10MB3321.namprd10.prod.outlook.com (2603:10b6:5:1a7::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.24; Fri, 6 May 2022 17:56:10 +0000 Received: from DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::e81e:38af:cb6e:59e5]) by DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::e81e:38af:cb6e:59e5%4]) with mapi id 15.20.5227.020; Fri, 6 May 2022 17:56:10 +0000 Message-ID: <85bd80b4-b4fd-0d3f-a2e5-149559f2f387@oracle.com> Date: Fri, 6 May 2022 10:56:07 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.8.0 Subject: Re: [PATCH 2/3] mm: rmap: Fix CONT-PTE/PMD size hugetlb issue when migration Content-Language: en-US To: Baolin Wang , akpm@linux-foundation.org, catalin.marinas@arm.com, will@kernel.org Cc: tsbogend@alpha.franken.de, James.Bottomley@HansenPartnership.com, deller@gmx.de, mpe@ellerman.id.au, benh@kernel.crashing.org, paulus@samba.org, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, ysato@users.sourceforge.jp, dalias@libc.org, davem@davemloft.net, arnd@arndb.de, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-ia64@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org References: <11b92502b3df0e0bba6a1dc71476d79cab6c79ba.1651216964.git.baolin.wang@linux.alibaba.com> <5cab0eca-9630-a7c6-4f5d-5cb45ff82c83@oracle.com> <21b11024-e893-8c11-9b98-ab1d13413b61@linux.alibaba.com> From: Mike Kravetz In-Reply-To: <21b11024-e893-8c11-9b98-ab1d13413b61@linux.alibaba.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-ClientProxiedBy: MW4P220CA0018.NAMP220.PROD.OUTLOOK.COM (2603:10b6:303:115::23) To DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c42a7592-f19c-49c3-a51c-08da2f89b423 X-MS-TrafficTypeDiagnostic: DM6PR10MB3321:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: zjKhOp3nWFz4YdOCaPRRB29ZRh7650JBF6YgVueZCZ9hOp8i+4hhbSmtohz6NxsIrCYgbg4UIoKxJQEABny3dxJx9fzkl9lwr7ookOEZB/H10gxyVVpWLHb3aA3jh8wLFaVg73yqU5vbqzRb6DbJAKGHz/NkgTSt+WcVGLaRcdumfHuQdsE6I5EjXTgSjEVQXH8jUyrOjpD+erGl/mMWAv47G/2iYqdVu5OD4RgtQv81qGr9VSmPZjQdN1QGLnpTgmAlC4qrn+92gLiUBlggUQ6EOxW1JLn5FOHxWA2TCTslYJdhZILh5KsLAHMCgDGjddTpU+tKzG8vaNJdFXh02V+bZEmPSm58lpP7wSKZt0lClfGWJ0ujwGpcs9Ny5OsJQ5YI7odL9Ix9FbRtOZeXJVec51AU8uspssF03N1kvYDmxblDI+8tZz9wXsQamQN83rN9Y6bTR75P8VGwrNDxdJ/NTdWQSgVXyantfRiFuLfMt8TyoA0+2OXjn1IFiBF6i1XXqbnuNmYIg+guQbZ7dcYPOUDx+s2K/pvpjMRgasdnKdh1CqdLxyLm1o/nzAP2ArpFkcEPbOMG2NUQc8lj+RUaybJH3XQ9sdK/Plpi34V/KxCzL7X7a+eCo50NOv26/EBh2CVTKmTQGeLrrOmp/UkRSfcZhri9L7bhI1Hv+bwea7wTz4LpZtMkdoG5kY1JUunUtja63ENWHvLzCY24lACpahgs4kuC6SBQcuVVtZRvdJK3tj9pI4DiiRjuz929GbuuBq05lkTrydnLlw8s+g== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4201.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(53546011)(26005)(6506007)(2906002)(6666004)(83380400001)(52116002)(2616005)(186003)(6512007)(4326008)(316002)(5660300002)(36756003)(7416002)(7406005)(86362001)(31686004)(508600001)(6486002)(8936002)(38350700002)(38100700002)(31696002)(44832011)(66556008)(66476007)(66946007)(8676002)(43740500002)(45980500001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bVBJeEJQZklNSVpPNGNSaUxWNXV4elgxUGZBazhibC9LeUYzMnFHVHFiRzc0?= =?utf-8?B?bWZTc2t0VlVQL2hFc1hOMWFSWG5IY2pJRSs1UlV4MjBUK0lBT1pSczJUNGMz?= =?utf-8?B?OUlYMktuajJJQnRkdUE4aVlHb1NlNHp4dFIxSHV0aGFWVGFTNWt5UnFTYy92?= =?utf-8?B?T0krTlZHamkycTYrLzN0c1h3U3RyTkJwSDUrazFLZDZieVhTemYrWVJZck4z?= =?utf-8?B?c2lJcG40RnMrRWFEWGJQbytBd2ZKN1NaZklKR09jSENvVEM3S3ViK2tEdUd6?= =?utf-8?B?T3U0cjBHM1B0U2VqVys5M3hCZHB6TGRKdmY3b08yajJhMkhmNm1DNFhBR1JX?= =?utf-8?B?WTRadVUzeWdTVkVZaUtPbmZjeEdJNmhpcFVJeFg4ZTEyRVlRd3ErVXhXMW1X?= =?utf-8?B?NU9xZEZ0ZXg5ZExJbHkybDNDblo1dkE4T3FaejR0Z3I3Yk9ZdGFVTUt1cVhJ?= =?utf-8?B?b3RpVWZSYmJnZm9TNmxEenhiQWlaQ01TWmNReFgwZHBoUEdGdWJOS3VIWlVY?= =?utf-8?B?NUEzUkZ4ejFQdnhUUEJCNEV3ZUNBZCtXN0Z2SUxZWDFEd09RZDgwVlhvbWw0?= =?utf-8?B?VEdQUWRWUFFRYngwdWhiaE1yVGVtZXRHdjNiUWlpL3lBUnFpY2orMmRQd2I1?= =?utf-8?B?aXRabzUzbzVLbncrWHFYS2NxT2FRWlhXbEVEdFpmeVVkUk1xRjhVTVRpSFEv?= =?utf-8?B?aEg2cmQ4NzV2RlJpdzAyVVA5bGZXTngyTFY0Uk83UXkzTGZ5OFMzTGNBL1FL?= =?utf-8?B?NmdsZTV3RFhzYnlTT2xBZXhFRzAzY3llYitwRVJhTGxhaFBhVEZMMnNERmhl?= =?utf-8?B?TWhNNS9lZU1mY2Q2SW42VWt1RnBzdDBZaGhoelMxU1RBY21RWnBVVUo1TDJw?= =?utf-8?B?ajYvSTBnL3FqYTRJWmlLMFFvQlRwRUNZRE81Z2dNbzQrZFdqSGNHSkdabVg0?= =?utf-8?B?ci83REsrRGRPOGc5d1U1YW1JRkppZTRldE83clI0NURiOHlPZytxMnN3NnZs?= =?utf-8?B?MTNLZVdaYzQ3ckFrTWJoVncxZEVkVk1Ua3c2ODZEa1Jvb0dkd2gwZGV3alZV?= =?utf-8?B?VVErVjJscm9KVUFwWStMSUdONlRwRGg3S1dOaHVqOFJNRVhBMXVTZVhWY3JP?= =?utf-8?B?TzZYaWJYWTNQcy9PMnR0M3AyS3pIeGZoQ1VHNVNuMVlHRWtWTDFWdUh1VHlo?= =?utf-8?B?aERUZWViaHNDRDh5NnB4OWxYc0ZUTERESlF5SUU3SUpUWnBjRVZqZjZaQjlI?= =?utf-8?B?NTZtNFpRdDdFRDhFb25QVHZCNlE2SFhwd2t0eURJRi96MHgrYzZJNXdFb3hU?= =?utf-8?B?RjFnbllTK1M2aFcwT1ZDdStjZjJncFFwSHp0WEdwUG1TWnhHL01tcE9kRVA3?= =?utf-8?B?am5PVmY3ZWpMVVdXQ1dUeHRiV25IWVNHNlhkYS9DWHVZTUt1VExBRTRNbWRC?= =?utf-8?B?S0E4WFZlcW1Td3I3ZVVRblhqditLRDN6UTBQNTBucG04anVIbkhLcmpSRklB?= =?utf-8?B?RUI1UFJVN1FNbFgvZkRrY0tRYWNSNGt4eG9pZ2hLOGRBbVFvdUhaNlRrWkIy?= =?utf-8?B?T25GUTYwWE1ubnVrMHJnUzZpT2NmT0hMSU1qaVRnbnRQM3liYTV3N1BoMnVM?= =?utf-8?B?QVRsa1lBc0NkZWlibHNZRFlNYWVPcEMyNGJwQ21ZTU1yVDA5OExsQnZOaDlS?= =?utf-8?B?c2pYYjFjRFg3aUFvYnliNlloSzIxSnBZTGt6emFOemQvMm9DL1FpeDMzVHR5?= =?utf-8?B?UFQ0UkE5ZzlEbG1Hb2liYnFLblZ1SGhJR2FweXI0ZUlFUmpSQ2xta2Z4eHRs?= =?utf-8?B?RmdkWGl5UU9taWpOVkI0UjBTcC9MQVBaOVJhRTJiRG1zOGRWLytlTU44cW50?= =?utf-8?B?bGI2VGdrS0l3UkxNOFBUdk53VnU0Rmt4bnBKVWpXc1UxSDUzUldNMmQ1akwv?= =?utf-8?B?bVd2YzZTUmFtenJPcVYyRURqS1NOdERiSThsU2tqeC9EWElyK1BGU053c1g0?= =?utf-8?B?QkFhV1dWbklFQ002MEo3S3NndzIxdmtZZjQ2aVFYVFkzRHFFSlhhMTlyZDZt?= =?utf-8?B?cFo4UG1uMU8zcEF2VGIvSlRXWGdBdGd0c0c0YWRtSXJGVGZaR2JMZklrQjVi?= =?utf-8?B?cERraVdDbi9CQjY2TFhNSVhTZ2RBWnlVWFYrK1RibnFydDdOSUxha1NxV1Na?= =?utf-8?B?d28rYlZPTlJQYmVLUzI2b2E2aTZ2aWpyOFZIb095MFVBRDY4NTV3QUU1Z0Nm?= =?utf-8?B?Vm44VWpLN3FRQ3hON0F1RDlHQzdkTkpCb1FaUUpKdDVwZjNWM2ljQ05DV1d6?= =?utf-8?B?dGVpZndiemNqd09UNkxzejQxOVFaemNlNkp5dDdXWjJsWUZRa3pOMjVKVlNp?= =?utf-8?Q?HguP9yiBHDzAad78=3D?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: c42a7592-f19c-49c3-a51c-08da2f89b423 X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4201.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 May 2022 17:56:10.7793 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: XbKCDCkv5ffaYXs43rx2K4D3gvl9Z9W4Jll47blD0Y0FpxpJ/Dkvg0fG3DaE9jLohinFWXfZxZY/Ld8nYP07mg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR10MB3321 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486,18.0.858 definitions=2022-05-06_07:2022-05-05,2022-05-06 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 malwarescore=0 spamscore=0 mlxlogscore=845 mlxscore=0 suspectscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205060091 X-Proofpoint-ORIG-GUID: aKof-JxTZ_IL3JOHP418iQRiEd6Ec4Ty X-Proofpoint-GUID: aKof-JxTZ_IL3JOHP418iQRiEd6Ec4Ty Precedence: bulk List-ID: X-Mailing-List: linux-sh@vger.kernel.org On 5/5/22 20:39, Baolin Wang wrote: > > On 5/6/2022 7:53 AM, Mike Kravetz wrote: >> On 4/29/22 01:14, Baolin Wang wrote: >>> On some architectures (like ARM64), it can support CONT-PTE/PMD size >>> hugetlb, which means it can support not only PMD/PUD size hugetlb: >>> 2M and 1G, but also CONT-PTE/PMD size: 64K and 32M if a 4K page >>> size specified. >> >>> diff --git a/mm/rmap.c b/mm/rmap.c >>> index 6fdd198..7cf2408 100644 >>> --- a/mm/rmap.c >>> +++ b/mm/rmap.c >>> @@ -1924,13 +1924,15 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, >>>                       break; >>>                   } >>>               } >>> + >>> +            /* Nuke the hugetlb page table entry */ >>> +            pteval = huge_ptep_clear_flush(vma, address, pvmw.pte); >>>           } else { >>>               flush_cache_page(vma, address, pte_pfn(*pvmw.pte)); >>> +            /* Nuke the page table entry. */ >>> +            pteval = ptep_clear_flush(vma, address, pvmw.pte); >>>           } >>>   >> >> On arm64 with CONT-PTE/PMD the returned pteval will have dirty or young set >> if ANY of the PTE/PMDs had dirty or young set. > > Right. > >> >>> -        /* Nuke the page table entry. */ >>> -        pteval = ptep_clear_flush(vma, address, pvmw.pte); >>> - >>>           /* Set the dirty flag on the folio now the pte is gone. */ >>>           if (pte_dirty(pteval)) >>>               folio_mark_dirty(folio); >>> @@ -2015,7 +2017,10 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, >>>               pte_t swp_pte; >>>                 if (arch_unmap_one(mm, vma, address, pteval) < 0) { >>> -                set_pte_at(mm, address, pvmw.pte, pteval); >>> +                if (folio_test_hugetlb(folio)) >>> +                    set_huge_pte_at(mm, address, pvmw.pte, pteval); >> >> And, we will use that pteval for ALL the PTE/PMDs here.  So, we would set >> the dirty or young bit in ALL PTE/PMDs. >> >> Could that cause any issues?  May be more of a question for the arm64 people. > > I don't think this will cause any issues. Since the hugetlb can not be split, and we should not lose the the dirty or young state if any subpages were set. Meanwhile we already did like this in hugetlb.c: > > pte = huge_ptep_get_and_clear(mm, address, ptep); > tlb_remove_huge_tlb_entry(h, tlb, ptep, address); > if (huge_pte_dirty(pte)) >     set_page_dirty(page); > Agree that it 'should not' cause issues. It just seems inconsistent. This is not a problem specifically with your patch, just the handling of CONT-PTE/PMD entries. There does not appear to be an arm64 specific version of huge_ptep_get() that takes CONT-PTE/PMD into account. So, huge_ptep_get() would only return the one specific value. It would not take into account the dirty or young bits of CONT-PTE/PMDs like your new version of huge_ptep_get_and_clear. Is that correct? Or, am I missing something. If I am correct, then code like the following may not work: static int gather_hugetlb_stats(pte_t *pte, unsigned long hmask, unsigned long addr, unsigned long end, struct mm_walk *walk) { pte_t huge_pte = huge_ptep_get(pte); struct numa_maps *md; struct page *page; if (!pte_present(huge_pte)) return 0; page = pte_page(huge_pte); md = walk->private; gather_stats(page, md, pte_dirty(huge_pte), 1); return 0; } -- Mike Kravetz From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mike Kravetz Date: Fri, 06 May 2022 17:56:07 +0000 Subject: Re: [PATCH 2/3] mm: rmap: Fix CONT-PTE/PMD size hugetlb issue when migration Message-Id: <85bd80b4-b4fd-0d3f-a2e5-149559f2f387@oracle.com> List-Id: References: <11b92502b3df0e0bba6a1dc71476d79cab6c79ba.1651216964.git.baolin.wang@linux.alibaba.com> <5cab0eca-9630-a7c6-4f5d-5cb45ff82c83@oracle.com> <21b11024-e893-8c11-9b98-ab1d13413b61@linux.alibaba.com> In-Reply-To: <21b11024-e893-8c11-9b98-ab1d13413b61@linux.alibaba.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit To: Baolin Wang , akpm@linux-foundation.org, catalin.marinas@arm.com, will@kernel.org Cc: tsbogend@alpha.franken.de, James.Bottomley@HansenPartnership.com, deller@gmx.de, mpe@ellerman.id.au, benh@kernel.crashing.org, paulus@samba.org, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, ysato@users.sourceforge.jp, dalias@libc.org, davem@davemloft.net, arnd@arndb.de, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-ia64@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org On 5/5/22 20:39, Baolin Wang wrote: > > On 5/6/2022 7:53 AM, Mike Kravetz wrote: >> On 4/29/22 01:14, Baolin Wang wrote: >>> On some architectures (like ARM64), it can support CONT-PTE/PMD size >>> hugetlb, which means it can support not only PMD/PUD size hugetlb: >>> 2M and 1G, but also CONT-PTE/PMD size: 64K and 32M if a 4K page >>> size specified. >> >>> diff --git a/mm/rmap.c b/mm/rmap.c >>> index 6fdd198..7cf2408 100644 >>> --- a/mm/rmap.c >>> +++ b/mm/rmap.c >>> @@ -1924,13 +1924,15 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, >>>                       break; >>>                   } >>>               } >>> + >>> +            /* Nuke the hugetlb page table entry */ >>> +            pteval = huge_ptep_clear_flush(vma, address, pvmw.pte); >>>           } else { >>>               flush_cache_page(vma, address, pte_pfn(*pvmw.pte)); >>> +            /* Nuke the page table entry. */ >>> +            pteval = ptep_clear_flush(vma, address, pvmw.pte); >>>           } >>>   >> >> On arm64 with CONT-PTE/PMD the returned pteval will have dirty or young set >> if ANY of the PTE/PMDs had dirty or young set. > > Right. > >> >>> -        /* Nuke the page table entry. */ >>> -        pteval = ptep_clear_flush(vma, address, pvmw.pte); >>> - >>>           /* Set the dirty flag on the folio now the pte is gone. */ >>>           if (pte_dirty(pteval)) >>>               folio_mark_dirty(folio); >>> @@ -2015,7 +2017,10 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, >>>               pte_t swp_pte; >>>                 if (arch_unmap_one(mm, vma, address, pteval) < 0) { >>> -                set_pte_at(mm, address, pvmw.pte, pteval); >>> +                if (folio_test_hugetlb(folio)) >>> +                    set_huge_pte_at(mm, address, pvmw.pte, pteval); >> >> And, we will use that pteval for ALL the PTE/PMDs here.  So, we would set >> the dirty or young bit in ALL PTE/PMDs. >> >> Could that cause any issues?  May be more of a question for the arm64 people. > > I don't think this will cause any issues. Since the hugetlb can not be split, and we should not lose the the dirty or young state if any subpages were set. Meanwhile we already did like this in hugetlb.c: > > pte = huge_ptep_get_and_clear(mm, address, ptep); > tlb_remove_huge_tlb_entry(h, tlb, ptep, address); > if (huge_pte_dirty(pte)) >     set_page_dirty(page); > Agree that it 'should not' cause issues. It just seems inconsistent. This is not a problem specifically with your patch, just the handling of CONT-PTE/PMD entries. There does not appear to be an arm64 specific version of huge_ptep_get() that takes CONT-PTE/PMD into account. So, huge_ptep_get() would only return the one specific value. It would not take into account the dirty or young bits of CONT-PTE/PMDs like your new version of huge_ptep_get_and_clear. Is that correct? Or, am I missing something. If I am correct, then code like the following may not work: static int gather_hugetlb_stats(pte_t *pte, unsigned long hmask, unsigned long addr, unsigned long end, struct mm_walk *walk) { pte_t huge_pte = huge_ptep_get(pte); struct numa_maps *md; struct page *page; if (!pte_present(huge_pte)) return 0; page = pte_page(huge_pte); md = walk->private; gather_stats(page, md, pte_dirty(huge_pte), 1); return 0; } -- Mike Kravetz From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 75025C433F5 for ; Sat, 7 May 2022 09:00:02 +0000 (UTC) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4KwLww5rvZz3c8r for ; Sat, 7 May 2022 19:00:00 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=oracle.com header.i=@oracle.com header.a=rsa-sha256 header.s=corp-2021-07-09 header.b=szHQF+8l; dkim=pass (1024-bit key; unprotected) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.a=rsa-sha256 header.s=selector2-oracle-onmicrosoft-com header.b=ijtx8ilc; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=oracle.com (client-ip=205.220.165.32; helo=mx0a-00069f02.pphosted.com; envelope-from=mike.kravetz@oracle.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=oracle.com header.i=@oracle.com header.a=rsa-sha256 header.s=corp-2021-07-09 header.b=szHQF+8l; dkim=pass (1024-bit key; unprotected) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.a=rsa-sha256 header.s=selector2-oracle-onmicrosoft-com header.b=ijtx8ilc; dkim-atps=neutral Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4KwLw32PbPz3bkv for ; Sat, 7 May 2022 18:59:12 +1000 (AEST) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 246FW2al030007; Fri, 6 May 2022 17:56:15 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=message-id : date : subject : to : cc : references : from : in-reply-to : content-type : content-transfer-encoding : mime-version; s=corp-2021-07-09; bh=dvzpIwt5+QcB24+EByvUA0dnLmNi8FJ99aXDZjGDbU8=; b=szHQF+8lv4TfY1qZu8KtHdXOMO9JhGkuR7v1rSdjMV4+gSaybmTsGN8hmPH/hygcgbzM wlUmNy7oBR8CI8OxlJtZvZwNSS9y6VYhUlnmcYdJmirqintgM+cAW8sIWHhmAEzl96ke Qg2gidSPDzMWZvbK/GtLGcaPxV5Cl0JwAVgP/V4FW9TqVBzAcjjJbhvwOKbLJk83q6vH fHALVF7as2aykVD8nao2EcEQPSBvBIlkSz9ZwY1kJPqxZZgRVeWjamfY9iKwC98Pqzrq RhlSyNVV0Lg0X6qXDT7qVr4T5e/IIKXBp/hCX+pCZD+0QlzaunN8EkNa30FYRslqUeUF CQ== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3fruq0pvvs-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 06 May 2022 17:56:14 +0000 Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 246HtSCU026597; Fri, 6 May 2022 17:56:13 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2175.outbound.protection.outlook.com [104.47.59.175]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3fs1a8k75y-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 06 May 2022 17:56:13 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=eGm4n8FtnVnsiGxgrxhhbwHGduLBBRYr+Tw430xh9UjHuFfX7f/CS+BlALtj5KKGculHUPaT4K2xioGdhU64Rw7B9PbmmtA9rDzD6GU7h2ZfHonm2lOWOxHEyE+zSu0r3iqgcye/c/8+eH4QoQU3ahYHy9oImlgCpXl/2Q1sdehLoxWKfihJMU7uvBDfyF3NAVVHrGMhlvgvFCQqu8WD1RRXXRoFn8BwXkBJB9TvxvF+IKk/E8u04F9/AXO0KbRXqHZ5Dw6fzeXpPZlAeXvqOD+NbMznrbOlV8DoX+x8TBC+/JCQxYv4mfknHOkn3fUGGuguAk85tDplEoI+d0mbsA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=dvzpIwt5+QcB24+EByvUA0dnLmNi8FJ99aXDZjGDbU8=; b=gxziFOax49XChajY/51gj+wxvP+YvY7/0L0RX58bwzaMj08zXZMpYcdVRRZ4MvcqljJYJ7W1SkEsWJIMjZdA4LuZRXMX+nK78+FUDV5xkmmX6rjXrWxzN/bEmbUQIuSOgCOqHGBWxQg1f/oDTCBdj3X36Y457r5XTPBSjFLI8NPs8J0UA52FWW5ZEIw+DECk4ClRK38X138Jt/nISd9PNEPdJkih40mSS1gJPEFBxkXhvnBbvlUUNlrCNpmYp5Rq+FVBOX+obQvXNHeB2eXLih2HNo9hmjatda7pQ/pBOHDtfoBeWRI5FzcsmzvVzBV9xArTUeIhMzdKaA8bnCIRHw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dvzpIwt5+QcB24+EByvUA0dnLmNi8FJ99aXDZjGDbU8=; b=ijtx8ilcoPusfnLBiELAw+1Q1dxZpPfgxAjUAl5LUHB4gJoUdMQQ7X8Rm4QPJl56ofP14pS5wJTjFvxLRwnTXGFa+/iV6qD3lKiS4QWEiPRnvevGkKqe6Q7S1iLAWhC92KbO+pvQob/9uvxo7gOXZMYydkc1WyfWflbt/B1CzVM= Received: from DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) by DM6PR10MB3321.namprd10.prod.outlook.com (2603:10b6:5:1a7::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.24; Fri, 6 May 2022 17:56:10 +0000 Received: from DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::e81e:38af:cb6e:59e5]) by DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::e81e:38af:cb6e:59e5%4]) with mapi id 15.20.5227.020; Fri, 6 May 2022 17:56:10 +0000 Message-ID: <85bd80b4-b4fd-0d3f-a2e5-149559f2f387@oracle.com> Date: Fri, 6 May 2022 10:56:07 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.8.0 Subject: Re: [PATCH 2/3] mm: rmap: Fix CONT-PTE/PMD size hugetlb issue when migration Content-Language: en-US To: Baolin Wang , akpm@linux-foundation.org, catalin.marinas@arm.com, will@kernel.org References: <11b92502b3df0e0bba6a1dc71476d79cab6c79ba.1651216964.git.baolin.wang@linux.alibaba.com> <5cab0eca-9630-a7c6-4f5d-5cb45ff82c83@oracle.com> <21b11024-e893-8c11-9b98-ab1d13413b61@linux.alibaba.com> From: Mike Kravetz In-Reply-To: <21b11024-e893-8c11-9b98-ab1d13413b61@linux.alibaba.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-ClientProxiedBy: MW4P220CA0018.NAMP220.PROD.OUTLOOK.COM (2603:10b6:303:115::23) To DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c42a7592-f19c-49c3-a51c-08da2f89b423 X-MS-TrafficTypeDiagnostic: DM6PR10MB3321:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: zjKhOp3nWFz4YdOCaPRRB29ZRh7650JBF6YgVueZCZ9hOp8i+4hhbSmtohz6NxsIrCYgbg4UIoKxJQEABny3dxJx9fzkl9lwr7ookOEZB/H10gxyVVpWLHb3aA3jh8wLFaVg73yqU5vbqzRb6DbJAKGHz/NkgTSt+WcVGLaRcdumfHuQdsE6I5EjXTgSjEVQXH8jUyrOjpD+erGl/mMWAv47G/2iYqdVu5OD4RgtQv81qGr9VSmPZjQdN1QGLnpTgmAlC4qrn+92gLiUBlggUQ6EOxW1JLn5FOHxWA2TCTslYJdhZILh5KsLAHMCgDGjddTpU+tKzG8vaNJdFXh02V+bZEmPSm58lpP7wSKZt0lClfGWJ0ujwGpcs9Ny5OsJQ5YI7odL9Ix9FbRtOZeXJVec51AU8uspssF03N1kvYDmxblDI+8tZz9wXsQamQN83rN9Y6bTR75P8VGwrNDxdJ/NTdWQSgVXyantfRiFuLfMt8TyoA0+2OXjn1IFiBF6i1XXqbnuNmYIg+guQbZ7dcYPOUDx+s2K/pvpjMRgasdnKdh1CqdLxyLm1o/nzAP2ArpFkcEPbOMG2NUQc8lj+RUaybJH3XQ9sdK/Plpi34V/KxCzL7X7a+eCo50NOv26/EBh2CVTKmTQGeLrrOmp/UkRSfcZhri9L7bhI1Hv+bwea7wTz4LpZtMkdoG5kY1JUunUtja63ENWHvLzCY24lACpahgs4kuC6SBQcuVVtZRvdJK3tj9pI4DiiRjuz929GbuuBq05lkTrydnLlw8s+g== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM6PR10MB4201.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(53546011)(26005)(6506007)(2906002)(6666004)(83380400001)(52116002)(2616005)(186003)(6512007)(4326008)(316002)(5660300002)(36756003)(7416002)(7406005)(86362001)(31686004)(508600001)(6486002)(8936002)(38350700002)(38100700002)(31696002)(44832011)(66556008)(66476007)(66946007)(8676002)(43740500002)(45980500001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bVBJeEJQZklNSVpPNGNSaUxWNXV4elgxUGZBazhibC9LeUYzMnFHVHFiRzc0?= =?utf-8?B?bWZTc2t0VlVQL2hFc1hOMWFSWG5IY2pJRSs1UlV4MjBUK0lBT1pSczJUNGMz?= =?utf-8?B?OUlYMktuajJJQnRkdUE4aVlHb1NlNHp4dFIxSHV0aGFWVGFTNWt5UnFTYy92?= =?utf-8?B?T0krTlZHamkycTYrLzN0c1h3U3RyTkJwSDUrazFLZDZieVhTemYrWVJZck4z?= =?utf-8?B?c2lJcG40RnMrRWFEWGJQbytBd2ZKN1NaZklKR09jSENvVEM3S3ViK2tEdUd6?= =?utf-8?B?T3U0cjBHM1B0U2VqVys5M3hCZHB6TGRKdmY3b08yajJhMkhmNm1DNFhBR1JX?= =?utf-8?B?WTRadVUzeWdTVkVZaUtPbmZjeEdJNmhpcFVJeFg4ZTEyRVlRd3ErVXhXMW1X?= =?utf-8?B?NU9xZEZ0ZXg5ZExJbHkybDNDblo1dkE4T3FaejR0Z3I3Yk9ZdGFVTUt1cVhJ?= =?utf-8?B?b3RpVWZSYmJnZm9TNmxEenhiQWlaQ01TWmNReFgwZHBoUEdGdWJOS3VIWlVY?= =?utf-8?B?NUEzUkZ4ejFQdnhUUEJCNEV3ZUNBZCtXN0Z2SUxZWDFEd09RZDgwVlhvbWw0?= =?utf-8?B?VEdQUWRWUFFRYngwdWhiaE1yVGVtZXRHdjNiUWlpL3lBUnFpY2orMmRQd2I1?= =?utf-8?B?aXRabzUzbzVLbncrWHFYS2NxT2FRWlhXbEVEdFpmeVVkUk1xRjhVTVRpSFEv?= =?utf-8?B?aEg2cmQ4NzV2RlJpdzAyVVA5bGZXTngyTFY0Uk83UXkzTGZ5OFMzTGNBL1FL?= =?utf-8?B?NmdsZTV3RFhzYnlTT2xBZXhFRzAzY3llYitwRVJhTGxhaFBhVEZMMnNERmhl?= =?utf-8?B?TWhNNS9lZU1mY2Q2SW42VWt1RnBzdDBZaGhoelMxU1RBY21RWnBVVUo1TDJw?= =?utf-8?B?ajYvSTBnL3FqYTRJWmlLMFFvQlRwRUNZRE81Z2dNbzQrZFdqSGNHSkdabVg0?= =?utf-8?B?ci83REsrRGRPOGc5d1U1YW1JRkppZTRldE83clI0NURiOHlPZytxMnN3NnZs?= =?utf-8?B?MTNLZVdaYzQ3ckFrTWJoVncxZEVkVk1Ua3c2ODZEa1Jvb0dkd2gwZGV3alZV?= =?utf-8?B?VVErVjJscm9KVUFwWStMSUdONlRwRGg3S1dOaHVqOFJNRVhBMXVTZVhWY3JP?= =?utf-8?B?TzZYaWJYWTNQcy9PMnR0M3AyS3pIeGZoQ1VHNVNuMVlHRWtWTDFWdUh1VHlo?= =?utf-8?B?aERUZWViaHNDRDh5NnB4OWxYc0ZUTERESlF5SUU3SUpUWnBjRVZqZjZaQjlI?= =?utf-8?B?NTZtNFpRdDdFRDhFb25QVHZCNlE2SFhwd2t0eURJRi96MHgrYzZJNXdFb3hU?= =?utf-8?B?RjFnbllTK1M2aFcwT1ZDdStjZjJncFFwSHp0WEdwUG1TWnhHL01tcE9kRVA3?= =?utf-8?B?am5PVmY3ZWpMVVdXQ1dUeHRiV25IWVNHNlhkYS9DWHVZTUt1VExBRTRNbWRC?= =?utf-8?B?S0E4WFZlcW1Td3I3ZVVRblhqditLRDN6UTBQNTBucG04anVIbkhLcmpSRklB?= =?utf-8?B?RUI1UFJVN1FNbFgvZkRrY0tRYWNSNGt4eG9pZ2hLOGRBbVFvdUhaNlRrWkIy?= =?utf-8?B?T25GUTYwWE1ubnVrMHJnUzZpT2NmT0hMSU1qaVRnbnRQM3liYTV3N1BoMnVM?= =?utf-8?B?QVRsa1lBc0NkZWlibHNZRFlNYWVPcEMyNGJwQ21ZTU1yVDA5OExsQnZOaDlS?= =?utf-8?B?c2pYYjFjRFg3aUFvYnliNlloSzIxSnBZTGt6emFOemQvMm9DL1FpeDMzVHR5?= =?utf-8?B?UFQ0UkE5ZzlEbG1Hb2liYnFLblZ1SGhJR2FweXI0ZUlFUmpSQ2xta2Z4eHRs?= =?utf-8?B?RmdkWGl5UU9taWpOVkI0UjBTcC9MQVBaOVJhRTJiRG1zOGRWLytlTU44cW50?= =?utf-8?B?bGI2VGdrS0l3UkxNOFBUdk53VnU0Rmt4bnBKVWpXc1UxSDUzUldNMmQ1akwv?= =?utf-8?B?bVd2YzZTUmFtenJPcVYyRURqS1NOdERiSThsU2tqeC9EWElyK1BGU053c1g0?= =?utf-8?B?QkFhV1dWbklFQ002MEo3S3NndzIxdmtZZjQ2aVFYVFkzRHFFSlhhMTlyZDZt?= =?utf-8?B?cFo4UG1uMU8zcEF2VGIvSlRXWGdBdGd0c0c0YWRtSXJGVGZaR2JMZklrQjVi?= =?utf-8?B?cERraVdDbi9CQjY2TFhNSVhTZ2RBWnlVWFYrK1RibnFydDdOSUxha1NxV1Na?= =?utf-8?B?d28rYlZPTlJQYmVLUzI2b2E2aTZ2aWpyOFZIb095MFVBRDY4NTV3QUU1Z0Nm?= =?utf-8?B?Vm44VWpLN3FRQ3hON0F1RDlHQzdkTkpCb1FaUUpKdDVwZjNWM2ljQ05DV1d6?= =?utf-8?B?dGVpZndiemNqd09UNkxzejQxOVFaemNlNkp5dDdXWjJsWUZRa3pOMjVKVlNp?= =?utf-8?Q?HguP9yiBHDzAad78=3D?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: c42a7592-f19c-49c3-a51c-08da2f89b423 X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4201.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 May 2022 17:56:10.7793 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: XbKCDCkv5ffaYXs43rx2K4D3gvl9Z9W4Jll47blD0Y0FpxpJ/Dkvg0fG3DaE9jLohinFWXfZxZY/Ld8nYP07mg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR10MB3321 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486, 18.0.858 definitions=2022-05-06_07:2022-05-05, 2022-05-06 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 malwarescore=0 spamscore=0 mlxlogscore=845 mlxscore=0 suspectscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205060091 X-Proofpoint-ORIG-GUID: aKof-JxTZ_IL3JOHP418iQRiEd6Ec4Ty X-Proofpoint-GUID: aKof-JxTZ_IL3JOHP418iQRiEd6Ec4Ty X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: dalias@libc.org, linux-ia64@vger.kernel.org, linux-sh@vger.kernel.org, linux-kernel@vger.kernel.org, James.Bottomley@HansenPartnership.com, linux-mm@kvack.org, paulus@samba.org, sparclinux@vger.kernel.org, agordeev@linux.ibm.com, linux-arch@vger.kernel.org, linux-s390@vger.kernel.org, arnd@arndb.de, ysato@users.sourceforge.jp, deller@gmx.de, borntraeger@linux.ibm.com, gor@linux.ibm.com, hca@linux.ibm.com, linux-arm-kernel@lists.infradead.org, tsbogend@alpha.franken.de, linux-parisc@vger.kernel.org, linux-mips@vger.kernel.org, svens@linux.ibm.com, linuxppc-dev@lists.ozlabs.org, davem@davemloft.net Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On 5/5/22 20:39, Baolin Wang wrote: > > On 5/6/2022 7:53 AM, Mike Kravetz wrote: >> On 4/29/22 01:14, Baolin Wang wrote: >>> On some architectures (like ARM64), it can support CONT-PTE/PMD size >>> hugetlb, which means it can support not only PMD/PUD size hugetlb: >>> 2M and 1G, but also CONT-PTE/PMD size: 64K and 32M if a 4K page >>> size specified. >> >>> diff --git a/mm/rmap.c b/mm/rmap.c >>> index 6fdd198..7cf2408 100644 >>> --- a/mm/rmap.c >>> +++ b/mm/rmap.c >>> @@ -1924,13 +1924,15 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, >>>                       break; >>>                   } >>>               } >>> + >>> +            /* Nuke the hugetlb page table entry */ >>> +            pteval = huge_ptep_clear_flush(vma, address, pvmw.pte); >>>           } else { >>>               flush_cache_page(vma, address, pte_pfn(*pvmw.pte)); >>> +            /* Nuke the page table entry. */ >>> +            pteval = ptep_clear_flush(vma, address, pvmw.pte); >>>           } >>>   >> >> On arm64 with CONT-PTE/PMD the returned pteval will have dirty or young set >> if ANY of the PTE/PMDs had dirty or young set. > > Right. > >> >>> -        /* Nuke the page table entry. */ >>> -        pteval = ptep_clear_flush(vma, address, pvmw.pte); >>> - >>>           /* Set the dirty flag on the folio now the pte is gone. */ >>>           if (pte_dirty(pteval)) >>>               folio_mark_dirty(folio); >>> @@ -2015,7 +2017,10 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, >>>               pte_t swp_pte; >>>                 if (arch_unmap_one(mm, vma, address, pteval) < 0) { >>> -                set_pte_at(mm, address, pvmw.pte, pteval); >>> +                if (folio_test_hugetlb(folio)) >>> +                    set_huge_pte_at(mm, address, pvmw.pte, pteval); >> >> And, we will use that pteval for ALL the PTE/PMDs here.  So, we would set >> the dirty or young bit in ALL PTE/PMDs. >> >> Could that cause any issues?  May be more of a question for the arm64 people. > > I don't think this will cause any issues. Since the hugetlb can not be split, and we should not lose the the dirty or young state if any subpages were set. Meanwhile we already did like this in hugetlb.c: > > pte = huge_ptep_get_and_clear(mm, address, ptep); > tlb_remove_huge_tlb_entry(h, tlb, ptep, address); > if (huge_pte_dirty(pte)) >     set_page_dirty(page); > Agree that it 'should not' cause issues. It just seems inconsistent. This is not a problem specifically with your patch, just the handling of CONT-PTE/PMD entries. There does not appear to be an arm64 specific version of huge_ptep_get() that takes CONT-PTE/PMD into account. So, huge_ptep_get() would only return the one specific value. It would not take into account the dirty or young bits of CONT-PTE/PMDs like your new version of huge_ptep_get_and_clear. Is that correct? Or, am I missing something. If I am correct, then code like the following may not work: static int gather_hugetlb_stats(pte_t *pte, unsigned long hmask, unsigned long addr, unsigned long end, struct mm_walk *walk) { pte_t huge_pte = huge_ptep_get(pte); struct numa_maps *md; struct page *page; if (!pte_present(huge_pte)) return 0; page = pte_page(huge_pte); md = walk->private; gather_stats(page, md, pte_dirty(huge_pte), 1); return 0; } -- Mike Kravetz From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 134CDC433EF for ; Fri, 6 May 2022 17:58:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:From: References:Cc:To:Subject:Date:Message-ID:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=1CEAO3a5CWpOkczgnhh+AwOAZJXXdGBIHHgyr90Bhfw=; b=GmcNAdYYYhuqOSEKXQNpLN968I QKgfFaccAc/9K7Z4Y9XISckg4jpbZr3+7DAekgC4B4810kzpD1aL6zZwG7ACUk0bTvvIidNZ2MYT5 ttoXhOq5TzIQDt0bIcHpfwdZO1gPv1NL+v/5Q+UAW/WgNSi5D4ha3/DVXk6iJAHIQ5sjXgoZKCA7b jLVdSkKGsal4AXTCT7BzV8twDNX+rYZIWHW5pQQbtVps9hAOhx/MGCW6wKQ5lbq3Q0C9opgsyfnvN ybH2AedGuauutJPL+6cq/5w4xs6rHqQ2JbZn4FUtvzaazlmF967eKMz6dE76EDaEtuzt2L/8X/Y30 JBPJev3A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nn2CW-004al0-F8; Fri, 06 May 2022 17:57:12 +0000 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nn2CT-004akd-1y for linux-arm-kernel@lists.infradead.org; Fri, 06 May 2022 17:57:10 +0000 Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 246FW2al030007; Fri, 6 May 2022 17:56:15 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=message-id : date : subject : to : cc : references : from : in-reply-to : content-type : content-transfer-encoding : mime-version; s=corp-2021-07-09; bh=dvzpIwt5+QcB24+EByvUA0dnLmNi8FJ99aXDZjGDbU8=; b=szHQF+8lv4TfY1qZu8KtHdXOMO9JhGkuR7v1rSdjMV4+gSaybmTsGN8hmPH/hygcgbzM wlUmNy7oBR8CI8OxlJtZvZwNSS9y6VYhUlnmcYdJmirqintgM+cAW8sIWHhmAEzl96ke Qg2gidSPDzMWZvbK/GtLGcaPxV5Cl0JwAVgP/V4FW9TqVBzAcjjJbhvwOKbLJk83q6vH fHALVF7as2aykVD8nao2EcEQPSBvBIlkSz9ZwY1kJPqxZZgRVeWjamfY9iKwC98Pqzrq RhlSyNVV0Lg0X6qXDT7qVr4T5e/IIKXBp/hCX+pCZD+0QlzaunN8EkNa30FYRslqUeUF CQ== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3fruq0pvvs-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 06 May 2022 17:56:14 +0000 Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 246HtSCU026597; Fri, 6 May 2022 17:56:13 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2175.outbound.protection.outlook.com [104.47.59.175]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3fs1a8k75y-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 06 May 2022 17:56:13 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=eGm4n8FtnVnsiGxgrxhhbwHGduLBBRYr+Tw430xh9UjHuFfX7f/CS+BlALtj5KKGculHUPaT4K2xioGdhU64Rw7B9PbmmtA9rDzD6GU7h2ZfHonm2lOWOxHEyE+zSu0r3iqgcye/c/8+eH4QoQU3ahYHy9oImlgCpXl/2Q1sdehLoxWKfihJMU7uvBDfyF3NAVVHrGMhlvgvFCQqu8WD1RRXXRoFn8BwXkBJB9TvxvF+IKk/E8u04F9/AXO0KbRXqHZ5Dw6fzeXpPZlAeXvqOD+NbMznrbOlV8DoX+x8TBC+/JCQxYv4mfknHOkn3fUGGuguAk85tDplEoI+d0mbsA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=dvzpIwt5+QcB24+EByvUA0dnLmNi8FJ99aXDZjGDbU8=; b=gxziFOax49XChajY/51gj+wxvP+YvY7/0L0RX58bwzaMj08zXZMpYcdVRRZ4MvcqljJYJ7W1SkEsWJIMjZdA4LuZRXMX+nK78+FUDV5xkmmX6rjXrWxzN/bEmbUQIuSOgCOqHGBWxQg1f/oDTCBdj3X36Y457r5XTPBSjFLI8NPs8J0UA52FWW5ZEIw+DECk4ClRK38X138Jt/nISd9PNEPdJkih40mSS1gJPEFBxkXhvnBbvlUUNlrCNpmYp5Rq+FVBOX+obQvXNHeB2eXLih2HNo9hmjatda7pQ/pBOHDtfoBeWRI5FzcsmzvVzBV9xArTUeIhMzdKaA8bnCIRHw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dvzpIwt5+QcB24+EByvUA0dnLmNi8FJ99aXDZjGDbU8=; b=ijtx8ilcoPusfnLBiELAw+1Q1dxZpPfgxAjUAl5LUHB4gJoUdMQQ7X8Rm4QPJl56ofP14pS5wJTjFvxLRwnTXGFa+/iV6qD3lKiS4QWEiPRnvevGkKqe6Q7S1iLAWhC92KbO+pvQob/9uvxo7gOXZMYydkc1WyfWflbt/B1CzVM= Received: from DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) by DM6PR10MB3321.namprd10.prod.outlook.com (2603:10b6:5:1a7::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.24; Fri, 6 May 2022 17:56:10 +0000 Received: from DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::e81e:38af:cb6e:59e5]) by DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::e81e:38af:cb6e:59e5%4]) with mapi id 15.20.5227.020; Fri, 6 May 2022 17:56:10 +0000 Message-ID: <85bd80b4-b4fd-0d3f-a2e5-149559f2f387@oracle.com> Date: Fri, 6 May 2022 10:56:07 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.8.0 Subject: Re: [PATCH 2/3] mm: rmap: Fix CONT-PTE/PMD size hugetlb issue when migration Content-Language: en-US To: Baolin Wang , akpm@linux-foundation.org, catalin.marinas@arm.com, will@kernel.org Cc: tsbogend@alpha.franken.de, James.Bottomley@HansenPartnership.com, deller@gmx.de, mpe@ellerman.id.au, benh@kernel.crashing.org, paulus@samba.org, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, ysato@users.sourceforge.jp, dalias@libc.org, davem@davemloft.net, arnd@arndb.de, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-ia64@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org References: <11b92502b3df0e0bba6a1dc71476d79cab6c79ba.1651216964.git.baolin.wang@linux.alibaba.com> <5cab0eca-9630-a7c6-4f5d-5cb45ff82c83@oracle.com> <21b11024-e893-8c11-9b98-ab1d13413b61@linux.alibaba.com> From: Mike Kravetz In-Reply-To: <21b11024-e893-8c11-9b98-ab1d13413b61@linux.alibaba.com> X-ClientProxiedBy: MW4P220CA0018.NAMP220.PROD.OUTLOOK.COM (2603:10b6:303:115::23) To DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c42a7592-f19c-49c3-a51c-08da2f89b423 X-MS-TrafficTypeDiagnostic: DM6PR10MB3321:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: zjKhOp3nWFz4YdOCaPRRB29ZRh7650JBF6YgVueZCZ9hOp8i+4hhbSmtohz6NxsIrCYgbg4UIoKxJQEABny3dxJx9fzkl9lwr7ookOEZB/H10gxyVVpWLHb3aA3jh8wLFaVg73yqU5vbqzRb6DbJAKGHz/NkgTSt+WcVGLaRcdumfHuQdsE6I5EjXTgSjEVQXH8jUyrOjpD+erGl/mMWAv47G/2iYqdVu5OD4RgtQv81qGr9VSmPZjQdN1QGLnpTgmAlC4qrn+92gLiUBlggUQ6EOxW1JLn5FOHxWA2TCTslYJdhZILh5KsLAHMCgDGjddTpU+tKzG8vaNJdFXh02V+bZEmPSm58lpP7wSKZt0lClfGWJ0ujwGpcs9Ny5OsJQ5YI7odL9Ix9FbRtOZeXJVec51AU8uspssF03N1kvYDmxblDI+8tZz9wXsQamQN83rN9Y6bTR75P8VGwrNDxdJ/NTdWQSgVXyantfRiFuLfMt8TyoA0+2OXjn1IFiBF6i1XXqbnuNmYIg+guQbZ7dcYPOUDx+s2K/pvpjMRgasdnKdh1CqdLxyLm1o/nzAP2ArpFkcEPbOMG2NUQc8lj+RUaybJH3XQ9sdK/Plpi34V/KxCzL7X7a+eCo50NOv26/EBh2CVTKmTQGeLrrOmp/UkRSfcZhri9L7bhI1Hv+bwea7wTz4LpZtMkdoG5kY1JUunUtja63ENWHvLzCY24lACpahgs4kuC6SBQcuVVtZRvdJK3tj9pI4DiiRjuz929GbuuBq05lkTrydnLlw8s+g== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM6PR10MB4201.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(53546011)(26005)(6506007)(2906002)(6666004)(83380400001)(52116002)(2616005)(186003)(6512007)(4326008)(316002)(5660300002)(36756003)(7416002)(7406005)(86362001)(31686004)(508600001)(6486002)(8936002)(38350700002)(38100700002)(31696002)(44832011)(66556008)(66476007)(66946007)(8676002)(43740500002)(45980500001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bVBJeEJQZklNSVpPNGNSaUxWNXV4elgxUGZBazhibC9LeUYzMnFHVHFiRzc0?= =?utf-8?B?bWZTc2t0VlVQL2hFc1hOMWFSWG5IY2pJRSs1UlV4MjBUK0lBT1pSczJUNGMz?= =?utf-8?B?OUlYMktuajJJQnRkdUE4aVlHb1NlNHp4dFIxSHV0aGFWVGFTNWt5UnFTYy92?= =?utf-8?B?T0krTlZHamkycTYrLzN0c1h3U3RyTkJwSDUrazFLZDZieVhTemYrWVJZck4z?= =?utf-8?B?c2lJcG40RnMrRWFEWGJQbytBd2ZKN1NaZklKR09jSENvVEM3S3ViK2tEdUd6?= =?utf-8?B?T3U0cjBHM1B0U2VqVys5M3hCZHB6TGRKdmY3b08yajJhMkhmNm1DNFhBR1JX?= =?utf-8?B?WTRadVUzeWdTVkVZaUtPbmZjeEdJNmhpcFVJeFg4ZTEyRVlRd3ErVXhXMW1X?= =?utf-8?B?NU9xZEZ0ZXg5ZExJbHkybDNDblo1dkE4T3FaejR0Z3I3Yk9ZdGFVTUt1cVhJ?= =?utf-8?B?b3RpVWZSYmJnZm9TNmxEenhiQWlaQ01TWmNReFgwZHBoUEdGdWJOS3VIWlVY?= =?utf-8?B?NUEzUkZ4ejFQdnhUUEJCNEV3ZUNBZCtXN0Z2SUxZWDFEd09RZDgwVlhvbWw0?= =?utf-8?B?VEdQUWRWUFFRYngwdWhiaE1yVGVtZXRHdjNiUWlpL3lBUnFpY2orMmRQd2I1?= =?utf-8?B?aXRabzUzbzVLbncrWHFYS2NxT2FRWlhXbEVEdFpmeVVkUk1xRjhVTVRpSFEv?= =?utf-8?B?aEg2cmQ4NzV2RlJpdzAyVVA5bGZXTngyTFY0Uk83UXkzTGZ5OFMzTGNBL1FL?= =?utf-8?B?NmdsZTV3RFhzYnlTT2xBZXhFRzAzY3llYitwRVJhTGxhaFBhVEZMMnNERmhl?= =?utf-8?B?TWhNNS9lZU1mY2Q2SW42VWt1RnBzdDBZaGhoelMxU1RBY21RWnBVVUo1TDJw?= =?utf-8?B?ajYvSTBnL3FqYTRJWmlLMFFvQlRwRUNZRE81Z2dNbzQrZFdqSGNHSkdabVg0?= =?utf-8?B?ci83REsrRGRPOGc5d1U1YW1JRkppZTRldE83clI0NURiOHlPZytxMnN3NnZs?= =?utf-8?B?MTNLZVdaYzQ3ckFrTWJoVncxZEVkVk1Ua3c2ODZEa1Jvb0dkd2gwZGV3alZV?= =?utf-8?B?VVErVjJscm9KVUFwWStMSUdONlRwRGg3S1dOaHVqOFJNRVhBMXVTZVhWY3JP?= =?utf-8?B?TzZYaWJYWTNQcy9PMnR0M3AyS3pIeGZoQ1VHNVNuMVlHRWtWTDFWdUh1VHlo?= =?utf-8?B?aERUZWViaHNDRDh5NnB4OWxYc0ZUTERESlF5SUU3SUpUWnBjRVZqZjZaQjlI?= =?utf-8?B?NTZtNFpRdDdFRDhFb25QVHZCNlE2SFhwd2t0eURJRi96MHgrYzZJNXdFb3hU?= =?utf-8?B?RjFnbllTK1M2aFcwT1ZDdStjZjJncFFwSHp0WEdwUG1TWnhHL01tcE9kRVA3?= =?utf-8?B?am5PVmY3ZWpMVVdXQ1dUeHRiV25IWVNHNlhkYS9DWHVZTUt1VExBRTRNbWRC?= =?utf-8?B?S0E4WFZlcW1Td3I3ZVVRblhqditLRDN6UTBQNTBucG04anVIbkhLcmpSRklB?= =?utf-8?B?RUI1UFJVN1FNbFgvZkRrY0tRYWNSNGt4eG9pZ2hLOGRBbVFvdUhaNlRrWkIy?= =?utf-8?B?T25GUTYwWE1ubnVrMHJnUzZpT2NmT0hMSU1qaVRnbnRQM3liYTV3N1BoMnVM?= =?utf-8?B?QVRsa1lBc0NkZWlibHNZRFlNYWVPcEMyNGJwQ21ZTU1yVDA5OExsQnZOaDlS?= =?utf-8?B?c2pYYjFjRFg3aUFvYnliNlloSzIxSnBZTGt6emFOemQvMm9DL1FpeDMzVHR5?= =?utf-8?B?UFQ0UkE5ZzlEbG1Hb2liYnFLblZ1SGhJR2FweXI0ZUlFUmpSQ2xta2Z4eHRs?= =?utf-8?B?RmdkWGl5UU9taWpOVkI0UjBTcC9MQVBaOVJhRTJiRG1zOGRWLytlTU44cW50?= =?utf-8?B?bGI2VGdrS0l3UkxNOFBUdk53VnU0Rmt4bnBKVWpXc1UxSDUzUldNMmQ1akwv?= =?utf-8?B?bVd2YzZTUmFtenJPcVYyRURqS1NOdERiSThsU2tqeC9EWElyK1BGU053c1g0?= =?utf-8?B?QkFhV1dWbklFQ002MEo3S3NndzIxdmtZZjQ2aVFYVFkzRHFFSlhhMTlyZDZt?= =?utf-8?B?cFo4UG1uMU8zcEF2VGIvSlRXWGdBdGd0c0c0YWRtSXJGVGZaR2JMZklrQjVi?= =?utf-8?B?cERraVdDbi9CQjY2TFhNSVhTZ2RBWnlVWFYrK1RibnFydDdOSUxha1NxV1Na?= =?utf-8?B?d28rYlZPTlJQYmVLUzI2b2E2aTZ2aWpyOFZIb095MFVBRDY4NTV3QUU1Z0Nm?= =?utf-8?B?Vm44VWpLN3FRQ3hON0F1RDlHQzdkTkpCb1FaUUpKdDVwZjNWM2ljQ05DV1d6?= =?utf-8?B?dGVpZndiemNqd09UNkxzejQxOVFaemNlNkp5dDdXWjJsWUZRa3pOMjVKVlNp?= =?utf-8?Q?HguP9yiBHDzAad78=3D?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: c42a7592-f19c-49c3-a51c-08da2f89b423 X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4201.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 May 2022 17:56:10.7793 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: XbKCDCkv5ffaYXs43rx2K4D3gvl9Z9W4Jll47blD0Y0FpxpJ/Dkvg0fG3DaE9jLohinFWXfZxZY/Ld8nYP07mg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR10MB3321 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486, 18.0.858 definitions=2022-05-06_07:2022-05-05, 2022-05-06 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 malwarescore=0 spamscore=0 mlxlogscore=845 mlxscore=0 suspectscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205060091 X-Proofpoint-ORIG-GUID: aKof-JxTZ_IL3JOHP418iQRiEd6Ec4Ty X-Proofpoint-GUID: aKof-JxTZ_IL3JOHP418iQRiEd6Ec4Ty X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220506_105709_151939_42D1A00B X-CRM114-Status: GOOD ( 28.13 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gNS81LzIyIDIwOjM5LCBCYW9saW4gV2FuZyB3cm90ZToKPiAKPiBPbiA1LzYvMjAyMiA3OjUz IEFNLCBNaWtlIEtyYXZldHogd3JvdGU6Cj4+IE9uIDQvMjkvMjIgMDE6MTQsIEJhb2xpbiBXYW5n IHdyb3RlOgo+Pj4gT24gc29tZSBhcmNoaXRlY3R1cmVzIChsaWtlIEFSTTY0KSwgaXQgY2FuIHN1 cHBvcnQgQ09OVC1QVEUvUE1EIHNpemUKPj4+IGh1Z2V0bGIsIHdoaWNoIG1lYW5zIGl0IGNhbiBz dXBwb3J0IG5vdCBvbmx5IFBNRC9QVUQgc2l6ZSBodWdldGxiOgo+Pj4gMk0gYW5kIDFHLCBidXQg YWxzbyBDT05ULVBURS9QTUQgc2l6ZTogNjRLIGFuZCAzMk0gaWYgYSA0SyBwYWdlCj4+PiBzaXpl IHNwZWNpZmllZC4KPj4gPHNuaXA+Cj4+PiBkaWZmIC0tZ2l0IGEvbW0vcm1hcC5jIGIvbW0vcm1h cC5jCj4+PiBpbmRleCA2ZmRkMTk4Li43Y2YyNDA4IDEwMDY0NAo+Pj4gLS0tIGEvbW0vcm1hcC5j Cj4+PiArKysgYi9tbS9ybWFwLmMKPj4+IEBAIC0xOTI0LDEzICsxOTI0LDE1IEBAIHN0YXRpYyBi b29sIHRyeV90b19taWdyYXRlX29uZShzdHJ1Y3QgZm9saW8gKmZvbGlvLCBzdHJ1Y3Qgdm1fYXJl YV9zdHJ1Y3QgKnZtYSwKPj4+IMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoCBicmVhazsKPj4+IMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgfQo+Pj4g wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgfQo+Pj4gKwo+Pj4gK8KgwqDCoMKgwqDCoMKgwqDC oMKgwqAgLyogTnVrZSB0aGUgaHVnZXRsYiBwYWdlIHRhYmxlIGVudHJ5ICovCj4+PiArwqDCoMKg wqDCoMKgwqDCoMKgwqDCoCBwdGV2YWwgPSBodWdlX3B0ZXBfY2xlYXJfZmx1c2godm1hLCBhZGRy ZXNzLCBwdm13LnB0ZSk7Cj4+PiDCoMKgwqDCoMKgwqDCoMKgwqAgfSBlbHNlIHsKPj4+IMKgwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgIGZsdXNoX2NhY2hlX3BhZ2Uodm1hLCBhZGRyZXNzLCBwdGVf cGZuKCpwdm13LnB0ZSkpOwo+Pj4gK8KgwqDCoMKgwqDCoMKgwqDCoMKgwqAgLyogTnVrZSB0aGUg cGFnZSB0YWJsZSBlbnRyeS4gKi8KPj4+ICvCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIHB0ZXZhbCA9 IHB0ZXBfY2xlYXJfZmx1c2godm1hLCBhZGRyZXNzLCBwdm13LnB0ZSk7Cj4+PiDCoMKgwqDCoMKg wqDCoMKgwqAgfQo+Pj4gwqAgCj4+Cj4+IE9uIGFybTY0IHdpdGggQ09OVC1QVEUvUE1EIHRoZSBy ZXR1cm5lZCBwdGV2YWwgd2lsbCBoYXZlIGRpcnR5IG9yIHlvdW5nIHNldAo+PiBpZiBBTlkgb2Yg dGhlIFBURS9QTURzIGhhZCBkaXJ0eSBvciB5b3VuZyBzZXQuCj4gCj4gUmlnaHQuCj4gCj4+Cj4+ PiAtwqDCoMKgwqDCoMKgwqAgLyogTnVrZSB0aGUgcGFnZSB0YWJsZSBlbnRyeS4gKi8KPj4+IC3C oMKgwqDCoMKgwqDCoCBwdGV2YWwgPSBwdGVwX2NsZWFyX2ZsdXNoKHZtYSwgYWRkcmVzcywgcHZt dy5wdGUpOwo+Pj4gLQo+Pj4gwqDCoMKgwqDCoMKgwqDCoMKgIC8qIFNldCB0aGUgZGlydHkgZmxh ZyBvbiB0aGUgZm9saW8gbm93IHRoZSBwdGUgaXMgZ29uZS4gKi8KPj4+IMKgwqDCoMKgwqDCoMKg wqDCoCBpZiAocHRlX2RpcnR5KHB0ZXZhbCkpCj4+PiDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oCBmb2xpb19tYXJrX2RpcnR5KGZvbGlvKTsKPj4+IEBAIC0yMDE1LDcgKzIwMTcsMTAgQEAgc3Rh dGljIGJvb2wgdHJ5X3RvX21pZ3JhdGVfb25lKHN0cnVjdCBmb2xpbyAqZm9saW8sIHN0cnVjdCB2 bV9hcmVhX3N0cnVjdCAqdm1hLAo+Pj4gwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgcHRlX3Qg c3dwX3B0ZTsKPj4+IMKgIMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIGlmIChhcmNoX3VubWFw X29uZShtbSwgdm1hLCBhZGRyZXNzLCBwdGV2YWwpIDwgMCkgewo+Pj4gLcKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoCBzZXRfcHRlX2F0KG1tLCBhZGRyZXNzLCBwdm13LnB0ZSwgcHRldmFs KTsKPj4+ICvCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgaWYgKGZvbGlvX3Rlc3RfaHVn ZXRsYihmb2xpbykpCj4+PiArwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAg c2V0X2h1Z2VfcHRlX2F0KG1tLCBhZGRyZXNzLCBwdm13LnB0ZSwgcHRldmFsKTsKPj4KPj4gQW5k LCB3ZSB3aWxsIHVzZSB0aGF0IHB0ZXZhbCBmb3IgQUxMIHRoZSBQVEUvUE1EcyBoZXJlLsKgIFNv LCB3ZSB3b3VsZCBzZXQKPj4gdGhlIGRpcnR5IG9yIHlvdW5nIGJpdCBpbiBBTEwgUFRFL1BNRHMu Cj4+Cj4+IENvdWxkIHRoYXQgY2F1c2UgYW55IGlzc3Vlcz/CoCBNYXkgYmUgbW9yZSBvZiBhIHF1 ZXN0aW9uIGZvciB0aGUgYXJtNjQgcGVvcGxlLgo+IAo+IEkgZG9uJ3QgdGhpbmsgdGhpcyB3aWxs IGNhdXNlIGFueSBpc3N1ZXMuIFNpbmNlIHRoZSBodWdldGxiIGNhbiBub3QgYmUgc3BsaXQsIGFu ZCB3ZSBzaG91bGQgbm90IGxvc2UgdGhlIHRoZSBkaXJ0eSBvciB5b3VuZyBzdGF0ZSBpZiBhbnkg c3VicGFnZXMgd2VyZSBzZXQuIE1lYW53aGlsZSB3ZSBhbHJlYWR5IGRpZCBsaWtlIHRoaXMgaW4g aHVnZXRsYi5jOgo+IAo+IHB0ZSA9IGh1Z2VfcHRlcF9nZXRfYW5kX2NsZWFyKG1tLCBhZGRyZXNz LCBwdGVwKTsKPiB0bGJfcmVtb3ZlX2h1Z2VfdGxiX2VudHJ5KGgsIHRsYiwgcHRlcCwgYWRkcmVz cyk7Cj4gaWYgKGh1Z2VfcHRlX2RpcnR5KHB0ZSkpCj4gwqDCoMKgwqBzZXRfcGFnZV9kaXJ0eShw YWdlKTsKPiAKCkFncmVlIHRoYXQgaXQgJ3Nob3VsZCBub3QnIGNhdXNlIGlzc3Vlcy4gIEl0IGp1 c3Qgc2VlbXMgaW5jb25zaXN0ZW50LgpUaGlzIGlzIG5vdCBhIHByb2JsZW0gc3BlY2lmaWNhbGx5 IHdpdGggeW91ciBwYXRjaCwganVzdCB0aGUgaGFuZGxpbmcgb2YKQ09OVC1QVEUvUE1EIGVudHJp ZXMuCgpUaGVyZSBkb2VzIG5vdCBhcHBlYXIgdG8gYmUgYW4gYXJtNjQgc3BlY2lmaWMgdmVyc2lv biBvZiBodWdlX3B0ZXBfZ2V0KCkKdGhhdCB0YWtlcyBDT05ULVBURS9QTUQgaW50byBhY2NvdW50 LiAgU28sIGh1Z2VfcHRlcF9nZXQoKSB3b3VsZCBvbmx5CnJldHVybiB0aGUgb25lIHNwZWNpZmlj IHZhbHVlLiAgSXQgd291bGQgbm90IHRha2UgaW50byBhY2NvdW50IHRoZSBkaXJ0eQpvciB5b3Vu ZyBiaXRzIG9mIENPTlQtUFRFL1BNRHMgbGlrZSB5b3VyIG5ldyB2ZXJzaW9uIG9mCmh1Z2VfcHRl cF9nZXRfYW5kX2NsZWFyLiAgSXMgdGhhdCBjb3JyZWN0PyAgT3IsIGFtIEkgbWlzc2luZyBzb21l dGhpbmcuCgpJZiBJIGFtIGNvcnJlY3QsIHRoZW4gY29kZSBsaWtlIHRoZSBmb2xsb3dpbmcgbWF5 IG5vdCB3b3JrOgoKc3RhdGljIGludCBnYXRoZXJfaHVnZXRsYl9zdGF0cyhwdGVfdCAqcHRlLCB1 bnNpZ25lZCBsb25nIGhtYXNrLAogICAgICAgICAgICAgICAgdW5zaWduZWQgbG9uZyBhZGRyLCB1 bnNpZ25lZCBsb25nIGVuZCwgc3RydWN0IG1tX3dhbGsgKndhbGspCnsKICAgICAgICBwdGVfdCBo dWdlX3B0ZSA9IGh1Z2VfcHRlcF9nZXQocHRlKTsKICAgICAgICBzdHJ1Y3QgbnVtYV9tYXBzICpt ZDsKICAgICAgICBzdHJ1Y3QgcGFnZSAqcGFnZTsKCiAgICAgICAgaWYgKCFwdGVfcHJlc2VudCho dWdlX3B0ZSkpCiAgICAgICAgICAgICAgICByZXR1cm4gMDsKCiAgICAgICAgcGFnZSA9IHB0ZV9w YWdlKGh1Z2VfcHRlKTsKCiAgICAgICAgbWQgPSB3YWxrLT5wcml2YXRlOwogICAgICAgIGdhdGhl cl9zdGF0cyhwYWdlLCBtZCwgcHRlX2RpcnR5KGh1Z2VfcHRlKSwgMSk7CiAgICAgICAgcmV0dXJu IDA7Cn0KCi0tIApNaWtlIEtyYXZldHoKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fCmxpbnV4LWFybS1rZXJuZWwgbWFpbGluZyBsaXN0CmxpbnV4LWFybS1r ZXJuZWxAbGlzdHMuaW5mcmFkZWFkLm9yZwpodHRwOi8vbGlzdHMuaW5mcmFkZWFkLm9yZy9tYWls bWFuL2xpc3RpbmZvL2xpbnV4LWFybS1rZXJuZWwK