From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 328E9C43461 for ; Fri, 11 Sep 2020 19:04:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C9D5C22208 for ; Fri, 11 Sep 2020 19:04:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="PAoqGBmI" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1725796AbgIKTEj (ORCPT ); Fri, 11 Sep 2020 15:04:39 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:38922 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725774AbgIKTEi (ORCPT ); Fri, 11 Sep 2020 15:04:38 -0400 Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 08BJ1nqJ070282; Fri, 11 Sep 2020 15:03:16 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=date : from : to : cc : subject : message-id : references : mime-version : content-type : content-transfer-encoding : in-reply-to; s=pp1; bh=BA+waGK444WFdAdGsxtmH4MVyQmh9mHMY6P6ku0v4Ks=; b=PAoqGBmI1WsEu/qH5mX1NCJjx4VjyBJDOUw68p7GuYHEFTEDLVyHklhTy+D05a9nhIEh S3OPtLLBBYAxRCIWYIGMEgVuSzV/wiY0aRinuDzcFKAICBrC4V6KZGYgqhBPlJ/HtW90 2sVdth6sFOm8qAfahbo8bWOSGUQUEITM97sVy3F3h16HR0GwbY8iTIlfBFmJ+neomaWW ttvdIJBuVK0LzRf3TnGmPJDMF9t2Br4ehU8orVA+6UbKHdmO/NDqhkIh6K8Prm4HrnX/ X/r9PwkYXTaBXZJhLWPSWh77gi2NbCByHKwD02OEDWOrr1Md8dXOnOojvEUCnbXgmBbM gw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 33geac150s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 11 Sep 2020 15:03:16 -0400 Received: from m0098394.ppops.net (m0098394.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 08BJ2G5J073420; Fri, 11 Sep 2020 15:03:15 -0400 Received: from ppma04ams.nl.ibm.com (63.31.33a9.ip4.static.sl-reverse.com [169.51.49.99]) by mx0a-001b2d01.pphosted.com with ESMTP id 33geac14yv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 11 Sep 2020 15:03:15 -0400 Received: from pps.filterd (ppma04ams.nl.ibm.com [127.0.0.1]) by ppma04ams.nl.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 08BIvA0f002964; Fri, 11 Sep 2020 19:03:12 GMT Received: from b06cxnps4074.portsmouth.uk.ibm.com (d06relay11.portsmouth.uk.ibm.com [9.149.109.196]) by ppma04ams.nl.ibm.com with ESMTP id 33c2a8fk1b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 11 Sep 2020 19:03:12 +0000 Received: from b06wcsmtp001.portsmouth.uk.ibm.com (b06wcsmtp001.portsmouth.uk.ibm.com [9.149.105.160]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 08BJ39xe22675840 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 11 Sep 2020 19:03:10 GMT Received: from b06wcsmtp001.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B6F01A405F; Fri, 11 Sep 2020 19:03:09 +0000 (GMT) Received: from b06wcsmtp001.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 02C6DA405B; Fri, 11 Sep 2020 19:03:08 +0000 (GMT) Received: from localhost (unknown [9.145.43.16]) by b06wcsmtp001.portsmouth.uk.ibm.com (Postfix) with ESMTPS; Fri, 11 Sep 2020 19:03:07 +0000 (GMT) Date: Fri, 11 Sep 2020 21:03:06 +0200 From: Vasily Gorbik To: Jason Gunthorpe , John Hubbard , Linus Torvalds Cc: Gerald Schaefer , Alexander Gordeev , Peter Zijlstra , Dave Hansen , LKML , linux-mm , linux-arch , Andrew Morton , Russell King , Mike Rapoport , Catalin Marinas , Will Deacon , Michael Ellerman , Benjamin Herrenschmidt , Paul Mackerras , Jeff Dike , Richard Weinberger , Dave Hansen , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Arnd Bergmann , Andrey Ryabinin , linux-x86 , linux-arm , linux-power , linux-sparc , linux-um , linux-s390 , Heiko Carstens , Christian Borntraeger , Claudio Imbrenda Subject: [PATCH] mm/gup: fix gup_fast with dynamic page table folding Message-ID: References: <20200911070939.GB1362448@hirez.programming.kicks-ass.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20200911070939.GB1362448@hirez.programming.kicks-ass.net> X-Patchwork-Bot: notify X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-09-11_09:2020-09-10,2020-09-11 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 impostorscore=0 lowpriorityscore=0 priorityscore=1501 malwarescore=0 mlxlogscore=999 phishscore=0 bulkscore=0 suspectscore=0 spamscore=0 adultscore=0 clxscore=1011 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2009110148 Sender: linux-arch-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-arch@vger.kernel.org Currently to make sure that every page table entry is read just once gup_fast walks perform READ_ONCE and pass pXd value down to the next gup_pXd_range function by value e.g.: static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) ... pudp = pud_offset(&p4d, addr); This function passes a reference on that local value copy to pXd_offset, and might get the very same pointer in return. This happens when the level is folded (on most arches), and that pointer should not be iterated. On s390 due to the fact that each task might have different 5,4 or 3-level address translation and hence different levels folded the logic is more complex and non-iteratable pointer to a local copy leads to severe problems. Here is an example of what happens with gup_fast on s390, for a task with 3-levels paging, crossing a 2 GB pud boundary: // addr = 0x1007ffff000, end = 0x10080001000 static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { unsigned long next; pud_t *pudp; // pud_offset returns &p4d itself (a pointer to a value on stack) pudp = pud_offset(&p4d, addr); do { // on second iteratation reading "random" stack value pud_t pud = READ_ONCE(*pudp); // next = 0x10080000000, due to PUD_SIZE/MASK != PGDIR_SIZE/MASK on s390 next = pud_addr_end(addr, end); ... } while (pudp++, addr = next, addr != end); // pudp++ iterating over stack return 1; } This happens since s390 moved to common gup code with commit d1874a0c2805 ("s390/mm: make the pxd_offset functions more robust") and commit 1a42010cdc26 ("s390/mm: convert to the generic get_user_pages_fast code"). s390 tried to mimic static level folding by changing pXd_offset primitives to always calculate top level page table offset in pgd_offset and just return the value passed when pXd_offset has to act as folded. What is crucial for gup_fast and what has been overlooked is that PxD_SIZE/MASK and thus pXd_addr_end should also change correspondingly. And the latter is not possible with dynamic folding. To fix the issue in addition to pXd values pass original pXdp pointers down to gup_pXd_range functions. And introduce pXd_offset_lockless helpers, which take an additional pXd entry value parameter. This has already been discussed in https://lkml.kernel.org/r/20190418100218.0a4afd51@mschwideX1 Cc: # 5.2+ Fixes: 1a42010cdc26 ("s390/mm: convert to the generic get_user_pages_fast code") Reviewed-by: Gerald Schaefer Reviewed-by: Alexander Gordeev Signed-off-by: Vasily Gorbik --- arch/s390/include/asm/pgtable.h | 42 +++++++++++++++++++++++---------- include/linux/pgtable.h | 10 ++++++++ mm/gup.c | 18 +++++++------- 3 files changed, 49 insertions(+), 21 deletions(-) diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h index 7eb01a5459cd..b55561cc8786 100644 --- a/arch/s390/include/asm/pgtable.h +++ b/arch/s390/include/asm/pgtable.h @@ -1260,26 +1260,44 @@ static inline pgd_t *pgd_offset_raw(pgd_t *pgd, unsigned long address) #define pgd_offset(mm, address) pgd_offset_raw(READ_ONCE((mm)->pgd), address) -static inline p4d_t *p4d_offset(pgd_t *pgd, unsigned long address) +static inline p4d_t *p4d_offset_lockless(pgd_t *pgdp, pgd_t pgd, unsigned long address) { - if ((pgd_val(*pgd) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R1) - return (p4d_t *) pgd_deref(*pgd) + p4d_index(address); - return (p4d_t *) pgd; + if ((pgd_val(pgd) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R1) + return (p4d_t *) pgd_deref(pgd) + p4d_index(address); + return (p4d_t *) pgdp; } +#define p4d_offset_lockless p4d_offset_lockless -static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address) +static inline p4d_t *p4d_offset(pgd_t *pgdp, unsigned long address) { - if ((p4d_val(*p4d) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R2) - return (pud_t *) p4d_deref(*p4d) + pud_index(address); - return (pud_t *) p4d; + return p4d_offset_lockless(pgdp, *pgdp, address); +} + +static inline pud_t *pud_offset_lockless(p4d_t *p4dp, p4d_t p4d, unsigned long address) +{ + if ((p4d_val(p4d) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R2) + return (pud_t *) p4d_deref(p4d) + pud_index(address); + return (pud_t *) p4dp; +} +#define pud_offset_lockless pud_offset_lockless + +static inline pud_t *pud_offset(p4d_t *p4dp, unsigned long address) +{ + return pud_offset_lockless(p4dp, *p4dp, address); } #define pud_offset pud_offset -static inline pmd_t *pmd_offset(pud_t *pud, unsigned long address) +static inline pmd_t *pmd_offset_lockless(pud_t *pudp, pud_t pud, unsigned long address) +{ + if ((pud_val(pud) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R3) + return (pmd_t *) pud_deref(pud) + pmd_index(address); + return (pmd_t *) pudp; +} +#define pmd_offset_lockless pmd_offset_lockless + +static inline pmd_t *pmd_offset(pud_t *pudp, unsigned long address) { - if ((pud_val(*pud) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R3) - return (pmd_t *) pud_deref(*pud) + pmd_index(address); - return (pmd_t *) pud; + return pmd_offset_lockless(pudp, *pudp, address); } #define pmd_offset pmd_offset diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index e8cbc2e795d5..e899d3506671 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1427,6 +1427,16 @@ typedef unsigned int pgtbl_mod_mask; #define mm_pmd_folded(mm) __is_defined(__PAGETABLE_PMD_FOLDED) #endif +#ifndef p4d_offset_lockless +#define p4d_offset_lockless(pgdp, pgd, address) p4d_offset(&pgd, address) +#endif +#ifndef pud_offset_lockless +#define pud_offset_lockless(p4dp, p4d, address) pud_offset(&p4d, address) +#endif +#ifndef pmd_offset_lockless +#define pmd_offset_lockless(pudp, pud, address) pmd_offset(&pud, address) +#endif + /* * p?d_leaf() - true if this entry is a final mapping to a physical address. * This differs from p?d_huge() by the fact that they are always available (if diff --git a/mm/gup.c b/mm/gup.c index e5739a1974d5..578bf5bd8bf8 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2485,13 +2485,13 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, return 1; } -static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end, +static int gup_pmd_range(pud_t *pudp, pud_t pud, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { unsigned long next; pmd_t *pmdp; - pmdp = pmd_offset(&pud, addr); + pmdp = pmd_offset_lockless(pudp, pud, addr); do { pmd_t pmd = READ_ONCE(*pmdp); @@ -2528,13 +2528,13 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end, return 1; } -static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, +static int gup_pud_range(p4d_t *p4dp, p4d_t p4d, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { unsigned long next; pud_t *pudp; - pudp = pud_offset(&p4d, addr); + pudp = pud_offset_lockless(p4dp, p4d, addr); do { pud_t pud = READ_ONCE(*pudp); @@ -2549,20 +2549,20 @@ static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, if (!gup_huge_pd(__hugepd(pud_val(pud)), addr, PUD_SHIFT, next, flags, pages, nr)) return 0; - } else if (!gup_pmd_range(pud, addr, next, flags, pages, nr)) + } else if (!gup_pmd_range(pudp, pud, addr, next, flags, pages, nr)) return 0; } while (pudp++, addr = next, addr != end); return 1; } -static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end, +static int gup_p4d_range(pgd_t *pgdp, pgd_t pgd, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { unsigned long next; p4d_t *p4dp; - p4dp = p4d_offset(&pgd, addr); + p4dp = p4d_offset_lockless(pgdp, pgd, addr); do { p4d_t p4d = READ_ONCE(*p4dp); @@ -2574,7 +2574,7 @@ static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end, if (!gup_huge_pd(__hugepd(p4d_val(p4d)), addr, P4D_SHIFT, next, flags, pages, nr)) return 0; - } else if (!gup_pud_range(p4d, addr, next, flags, pages, nr)) + } else if (!gup_pud_range(p4dp, p4d, addr, next, flags, pages, nr)) return 0; } while (p4dp++, addr = next, addr != end); @@ -2602,7 +2602,7 @@ static void gup_pgd_range(unsigned long addr, unsigned long end, if (!gup_huge_pd(__hugepd(pgd_val(pgd)), addr, PGDIR_SHIFT, next, flags, pages, nr)) return; - } else if (!gup_p4d_range(pgd, addr, next, flags, pages, nr)) + } else if (!gup_p4d_range(pgdp, pgd, addr, next, flags, pages, nr)) return; } while (pgdp++, addr = next, addr != end); } -- ⣿⣿⣿⣿⢋⡀⣀⠹⣿⣿⣿⣿ ⣿⣿⣿⣿⠠⣶⡦⠀⣿⣿⣿⣿ ⣿⣿⣿⠏⣴⣮⣴⣧⠈⢿⣿⣿ ⣿⣿⡏⢰⣿⠖⣠⣿⡆⠈⣿⣿ ⣿⢛⣵⣄⠙⣶⣶⡟⣅⣠⠹⣿ ⣿⣜⣛⠻⢎⣉⣉⣀⠿⣫⣵⣿ From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Date: Fri, 11 Sep 2020 21:03:06 +0200 From: Vasily Gorbik Subject: [PATCH] mm/gup: fix gup_fast with dynamic page table folding Message-ID: References: <20200911070939.GB1362448@hirez.programming.kicks-ass.net> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20200911070939.GB1362448@hirez.programming.kicks-ass.net> List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-um" Errors-To: linux-um-bounces+geert=linux-m68k.org@lists.infradead.org To: Jason Gunthorpe , John Hubbard , Linus Torvalds Cc: Peter Zijlstra , Benjamin Herrenschmidt , Dave Hansen , linux-mm , Paul Mackerras , linux-sparc , Alexander Gordeev , Claudio Imbrenda , Will Deacon , linux-arch , linux-s390 , Richard Weinberger , linux-x86 , Russell King , Christian Borntraeger , Ingo Molnar , Catalin Marinas , Andrey Ryabinin , Gerald Schaefer , Heiko Carstens , Arnd Bergmann , Jeff Dike , linux-um , Borislav Petkov , Andy Lutomirski , Thomas Gleixner , linux-arm , Dave Hansen , LKML , Michael Ellerman , Andrew Morton , linux-power , Mike Rapoport Q3VycmVudGx5IHRvIG1ha2Ugc3VyZSB0aGF0IGV2ZXJ5IHBhZ2UgdGFibGUgZW50cnkgaXMgcmVh ZCBqdXN0IG9uY2UKZ3VwX2Zhc3Qgd2Fsa3MgcGVyZm9ybSBSRUFEX09OQ0UgYW5kIHBhc3MgcFhk IHZhbHVlIGRvd24gdG8gdGhlIG5leHQKZ3VwX3BYZF9yYW5nZSBmdW5jdGlvbiBieSB2YWx1ZSBl LmcuOgoKc3RhdGljIGludCBndXBfcHVkX3JhbmdlKHA0ZF90IHA0ZCwgdW5zaWduZWQgbG9uZyBh ZGRyLCB1bnNpZ25lZCBsb25nIGVuZCwKICAgICAgICAgICAgICAgICAgICAgICAgIHVuc2lnbmVk IGludCBmbGFncywgc3RydWN0IHBhZ2UgKipwYWdlcywgaW50ICpucikKLi4uCiAgICAgICAgcHVk cCA9IHB1ZF9vZmZzZXQoJnA0ZCwgYWRkcik7CgpUaGlzIGZ1bmN0aW9uIHBhc3NlcyBhIHJlZmVy ZW5jZSBvbiB0aGF0IGxvY2FsIHZhbHVlIGNvcHkgdG8gcFhkX29mZnNldCwKYW5kIG1pZ2h0IGdl dCB0aGUgdmVyeSBzYW1lIHBvaW50ZXIgaW4gcmV0dXJuLiBUaGlzIGhhcHBlbnMgd2hlbiB0aGUK bGV2ZWwgaXMgZm9sZGVkIChvbiBtb3N0IGFyY2hlcyksIGFuZCB0aGF0IHBvaW50ZXIgc2hvdWxk IG5vdCBiZSBpdGVyYXRlZC4KCk9uIHMzOTAgZHVlIHRvIHRoZSBmYWN0IHRoYXQgZWFjaCB0YXNr IG1pZ2h0IGhhdmUgZGlmZmVyZW50IDUsNCBvcgozLWxldmVsIGFkZHJlc3MgdHJhbnNsYXRpb24g YW5kIGhlbmNlIGRpZmZlcmVudCBsZXZlbHMgZm9sZGVkIHRoZSBsb2dpYwppcyBtb3JlIGNvbXBs ZXggYW5kIG5vbi1pdGVyYXRhYmxlIHBvaW50ZXIgdG8gYSBsb2NhbCBjb3B5IGxlYWRzIHRvCnNl dmVyZSBwcm9ibGVtcy4KCkhlcmUgaXMgYW4gZXhhbXBsZSBvZiB3aGF0IGhhcHBlbnMgd2l0aCBn dXBfZmFzdCBvbiBzMzkwLCBmb3IgYSB0YXNrCndpdGggMy1sZXZlbHMgcGFnaW5nLCBjcm9zc2lu ZyBhIDIgR0IgcHVkIGJvdW5kYXJ5OgoKLy8gYWRkciA9IDB4MTAwN2ZmZmYwMDAsIGVuZCA9IDB4 MTAwODAwMDEwMDAKc3RhdGljIGludCBndXBfcHVkX3JhbmdlKHA0ZF90IHA0ZCwgdW5zaWduZWQg bG9uZyBhZGRyLCB1bnNpZ25lZCBsb25nIGVuZCwKICAgICAgICAgICAgICAgICAgICAgICAgIHVu c2lnbmVkIGludCBmbGFncywgc3RydWN0IHBhZ2UgKipwYWdlcywgaW50ICpucikKewogICAgICAg IHVuc2lnbmVkIGxvbmcgbmV4dDsKICAgICAgICBwdWRfdCAqcHVkcDsKCiAgICAgICAgLy8gcHVk X29mZnNldCByZXR1cm5zICZwNGQgaXRzZWxmIChhIHBvaW50ZXIgdG8gYSB2YWx1ZSBvbiBzdGFj aykKICAgICAgICBwdWRwID0gcHVkX29mZnNldCgmcDRkLCBhZGRyKTsKICAgICAgICBkbyB7CiAg ICAgICAgICAgICAgICAvLyBvbiBzZWNvbmQgaXRlcmF0YXRpb24gcmVhZGluZyAicmFuZG9tIiBz dGFjayB2YWx1ZQogICAgICAgICAgICAgICAgcHVkX3QgcHVkID0gUkVBRF9PTkNFKCpwdWRwKTsK CiAgICAgICAgICAgICAgICAvLyBuZXh0ID0gMHgxMDA4MDAwMDAwMCwgZHVlIHRvIFBVRF9TSVpF L01BU0sgIT0gUEdESVJfU0laRS9NQVNLIG9uIHMzOTAKICAgICAgICAgICAgICAgIG5leHQgPSBw dWRfYWRkcl9lbmQoYWRkciwgZW5kKTsKICAgICAgICAgICAgICAgIC4uLgogICAgICAgIH0gd2hp bGUgKHB1ZHArKywgYWRkciA9IG5leHQsIGFkZHIgIT0gZW5kKTsgLy8gcHVkcCsrIGl0ZXJhdGlu ZyBvdmVyIHN0YWNrCgogICAgICAgIHJldHVybiAxOwp9CgpUaGlzIGhhcHBlbnMgc2luY2UgczM5 MCBtb3ZlZCB0byBjb21tb24gZ3VwIGNvZGUgd2l0aApjb21taXQgZDE4NzRhMGMyODA1ICgiczM5 MC9tbTogbWFrZSB0aGUgcHhkX29mZnNldCBmdW5jdGlvbnMgbW9yZSByb2J1c3QiKQphbmQgY29t bWl0IDFhNDIwMTBjZGMyNiAoInMzOTAvbW06IGNvbnZlcnQgdG8gdGhlIGdlbmVyaWMKZ2V0X3Vz ZXJfcGFnZXNfZmFzdCBjb2RlIikuIHMzOTAgdHJpZWQgdG8gbWltaWMgc3RhdGljIGxldmVsIGZv bGRpbmcgYnkKY2hhbmdpbmcgcFhkX29mZnNldCBwcmltaXRpdmVzIHRvIGFsd2F5cyBjYWxjdWxh dGUgdG9wIGxldmVsIHBhZ2UgdGFibGUKb2Zmc2V0IGluIHBnZF9vZmZzZXQgYW5kIGp1c3QgcmV0 dXJuIHRoZSB2YWx1ZSBwYXNzZWQgd2hlbiBwWGRfb2Zmc2V0CmhhcyB0byBhY3QgYXMgZm9sZGVk LgoKV2hhdCBpcyBjcnVjaWFsIGZvciBndXBfZmFzdCBhbmQgd2hhdCBoYXMgYmVlbiBvdmVybG9v a2VkIGlzCnRoYXQgUHhEX1NJWkUvTUFTSyBhbmQgdGh1cyBwWGRfYWRkcl9lbmQgc2hvdWxkIGFs c28gY2hhbmdlCmNvcnJlc3BvbmRpbmdseS4gQW5kIHRoZSBsYXR0ZXIgaXMgbm90IHBvc3NpYmxl IHdpdGggZHluYW1pYyBmb2xkaW5nLgoKVG8gZml4IHRoZSBpc3N1ZSBpbiBhZGRpdGlvbiB0byBw WGQgdmFsdWVzIHBhc3Mgb3JpZ2luYWwKcFhkcCBwb2ludGVycyBkb3duIHRvIGd1cF9wWGRfcmFu Z2UgZnVuY3Rpb25zLiBBbmQgaW50cm9kdWNlCnBYZF9vZmZzZXRfbG9ja2xlc3MgaGVscGVycywg d2hpY2ggdGFrZSBhbiBhZGRpdGlvbmFsIHBYZAplbnRyeSB2YWx1ZSBwYXJhbWV0ZXIuIFRoaXMg aGFzIGFscmVhZHkgYmVlbiBkaXNjdXNzZWQgaW4KaHR0cHM6Ly9sa21sLmtlcm5lbC5vcmcvci8y MDE5MDQxODEwMDIxOC4wYTRhZmQ1MUBtc2Nod2lkZVgxCgpDYzogPHN0YWJsZUB2Z2VyLmtlcm5l bC5vcmc+ICMgNS4yKwpGaXhlczogMWE0MjAxMGNkYzI2ICgiczM5MC9tbTogY29udmVydCB0byB0 aGUgZ2VuZXJpYyBnZXRfdXNlcl9wYWdlc19mYXN0IGNvZGUiKQpSZXZpZXdlZC1ieTogR2VyYWxk IFNjaGFlZmVyIDxnZXJhbGQuc2NoYWVmZXJAbGludXguaWJtLmNvbT4KUmV2aWV3ZWQtYnk6IEFs ZXhhbmRlciBHb3JkZWV2IDxhZ29yZGVldkBsaW51eC5pYm0uY29tPgpTaWduZWQtb2ZmLWJ5OiBW YXNpbHkgR29yYmlrIDxnb3JAbGludXguaWJtLmNvbT4KLS0tCiBhcmNoL3MzOTAvaW5jbHVkZS9h c20vcGd0YWJsZS5oIHwgNDIgKysrKysrKysrKysrKysrKysrKysrKystLS0tLS0tLS0tCiBpbmNs dWRlL2xpbnV4L3BndGFibGUuaCAgICAgICAgIHwgMTAgKysrKysrKysKIG1tL2d1cC5jICAgICAg ICAgICAgICAgICAgICAgICAgfCAxOCArKysrKysrLS0tLS0tLQogMyBmaWxlcyBjaGFuZ2VkLCA0 OSBpbnNlcnRpb25zKCspLCAyMSBkZWxldGlvbnMoLSkKCmRpZmYgLS1naXQgYS9hcmNoL3MzOTAv aW5jbHVkZS9hc20vcGd0YWJsZS5oIGIvYXJjaC9zMzkwL2luY2x1ZGUvYXNtL3BndGFibGUuaApp bmRleCA3ZWIwMWE1NDU5Y2QuLmI1NTU2MWNjODc4NiAxMDA2NDQKLS0tIGEvYXJjaC9zMzkwL2lu Y2x1ZGUvYXNtL3BndGFibGUuaAorKysgYi9hcmNoL3MzOTAvaW5jbHVkZS9hc20vcGd0YWJsZS5o CkBAIC0xMjYwLDI2ICsxMjYwLDQ0IEBAIHN0YXRpYyBpbmxpbmUgcGdkX3QgKnBnZF9vZmZzZXRf cmF3KHBnZF90ICpwZ2QsIHVuc2lnbmVkIGxvbmcgYWRkcmVzcykKIAogI2RlZmluZSBwZ2Rfb2Zm c2V0KG1tLCBhZGRyZXNzKSBwZ2Rfb2Zmc2V0X3JhdyhSRUFEX09OQ0UoKG1tKS0+cGdkKSwgYWRk cmVzcykKIAotc3RhdGljIGlubGluZSBwNGRfdCAqcDRkX29mZnNldChwZ2RfdCAqcGdkLCB1bnNp Z25lZCBsb25nIGFkZHJlc3MpCitzdGF0aWMgaW5saW5lIHA0ZF90ICpwNGRfb2Zmc2V0X2xvY2ts ZXNzKHBnZF90ICpwZ2RwLCBwZ2RfdCBwZ2QsIHVuc2lnbmVkIGxvbmcgYWRkcmVzcykKIHsKLQlp ZiAoKHBnZF92YWwoKnBnZCkgJiBfUkVHSU9OX0VOVFJZX1RZUEVfTUFTSykgPj0gX1JFR0lPTl9F TlRSWV9UWVBFX1IxKQotCQlyZXR1cm4gKHA0ZF90ICopIHBnZF9kZXJlZigqcGdkKSArIHA0ZF9p bmRleChhZGRyZXNzKTsKLQlyZXR1cm4gKHA0ZF90ICopIHBnZDsKKwlpZiAoKHBnZF92YWwocGdk KSAmIF9SRUdJT05fRU5UUllfVFlQRV9NQVNLKSA+PSBfUkVHSU9OX0VOVFJZX1RZUEVfUjEpCisJ CXJldHVybiAocDRkX3QgKikgcGdkX2RlcmVmKHBnZCkgKyBwNGRfaW5kZXgoYWRkcmVzcyk7CisJ cmV0dXJuIChwNGRfdCAqKSBwZ2RwOwogfQorI2RlZmluZSBwNGRfb2Zmc2V0X2xvY2tsZXNzIHA0 ZF9vZmZzZXRfbG9ja2xlc3MKIAotc3RhdGljIGlubGluZSBwdWRfdCAqcHVkX29mZnNldChwNGRf dCAqcDRkLCB1bnNpZ25lZCBsb25nIGFkZHJlc3MpCitzdGF0aWMgaW5saW5lIHA0ZF90ICpwNGRf b2Zmc2V0KHBnZF90ICpwZ2RwLCB1bnNpZ25lZCBsb25nIGFkZHJlc3MpCiB7Ci0JaWYgKChwNGRf dmFsKCpwNGQpICYgX1JFR0lPTl9FTlRSWV9UWVBFX01BU0spID49IF9SRUdJT05fRU5UUllfVFlQ RV9SMikKLQkJcmV0dXJuIChwdWRfdCAqKSBwNGRfZGVyZWYoKnA0ZCkgKyBwdWRfaW5kZXgoYWRk cmVzcyk7Ci0JcmV0dXJuIChwdWRfdCAqKSBwNGQ7CisJcmV0dXJuIHA0ZF9vZmZzZXRfbG9ja2xl c3MocGdkcCwgKnBnZHAsIGFkZHJlc3MpOworfQorCitzdGF0aWMgaW5saW5lIHB1ZF90ICpwdWRf b2Zmc2V0X2xvY2tsZXNzKHA0ZF90ICpwNGRwLCBwNGRfdCBwNGQsIHVuc2lnbmVkIGxvbmcgYWRk cmVzcykKK3sKKwlpZiAoKHA0ZF92YWwocDRkKSAmIF9SRUdJT05fRU5UUllfVFlQRV9NQVNLKSA+ PSBfUkVHSU9OX0VOVFJZX1RZUEVfUjIpCisJCXJldHVybiAocHVkX3QgKikgcDRkX2RlcmVmKHA0 ZCkgKyBwdWRfaW5kZXgoYWRkcmVzcyk7CisJcmV0dXJuIChwdWRfdCAqKSBwNGRwOworfQorI2Rl ZmluZSBwdWRfb2Zmc2V0X2xvY2tsZXNzIHB1ZF9vZmZzZXRfbG9ja2xlc3MKKworc3RhdGljIGlu bGluZSBwdWRfdCAqcHVkX29mZnNldChwNGRfdCAqcDRkcCwgdW5zaWduZWQgbG9uZyBhZGRyZXNz KQoreworCXJldHVybiBwdWRfb2Zmc2V0X2xvY2tsZXNzKHA0ZHAsICpwNGRwLCBhZGRyZXNzKTsK IH0KICNkZWZpbmUgcHVkX29mZnNldCBwdWRfb2Zmc2V0CiAKLXN0YXRpYyBpbmxpbmUgcG1kX3Qg KnBtZF9vZmZzZXQocHVkX3QgKnB1ZCwgdW5zaWduZWQgbG9uZyBhZGRyZXNzKQorc3RhdGljIGlu bGluZSBwbWRfdCAqcG1kX29mZnNldF9sb2NrbGVzcyhwdWRfdCAqcHVkcCwgcHVkX3QgcHVkLCB1 bnNpZ25lZCBsb25nIGFkZHJlc3MpCit7CisJaWYgKChwdWRfdmFsKHB1ZCkgJiBfUkVHSU9OX0VO VFJZX1RZUEVfTUFTSykgPj0gX1JFR0lPTl9FTlRSWV9UWVBFX1IzKQorCQlyZXR1cm4gKHBtZF90 ICopIHB1ZF9kZXJlZihwdWQpICsgcG1kX2luZGV4KGFkZHJlc3MpOworCXJldHVybiAocG1kX3Qg KikgcHVkcDsKK30KKyNkZWZpbmUgcG1kX29mZnNldF9sb2NrbGVzcyBwbWRfb2Zmc2V0X2xvY2ts ZXNzCisKK3N0YXRpYyBpbmxpbmUgcG1kX3QgKnBtZF9vZmZzZXQocHVkX3QgKnB1ZHAsIHVuc2ln bmVkIGxvbmcgYWRkcmVzcykKIHsKLQlpZiAoKHB1ZF92YWwoKnB1ZCkgJiBfUkVHSU9OX0VOVFJZ X1RZUEVfTUFTSykgPj0gX1JFR0lPTl9FTlRSWV9UWVBFX1IzKQotCQlyZXR1cm4gKHBtZF90ICop IHB1ZF9kZXJlZigqcHVkKSArIHBtZF9pbmRleChhZGRyZXNzKTsKLQlyZXR1cm4gKHBtZF90ICop IHB1ZDsKKwlyZXR1cm4gcG1kX29mZnNldF9sb2NrbGVzcyhwdWRwLCAqcHVkcCwgYWRkcmVzcyk7 CiB9CiAjZGVmaW5lIHBtZF9vZmZzZXQgcG1kX29mZnNldAogCmRpZmYgLS1naXQgYS9pbmNsdWRl L2xpbnV4L3BndGFibGUuaCBiL2luY2x1ZGUvbGludXgvcGd0YWJsZS5oCmluZGV4IGU4Y2JjMmU3 OTVkNS4uZTg5OWQzNTA2NjcxIDEwMDY0NAotLS0gYS9pbmNsdWRlL2xpbnV4L3BndGFibGUuaAor KysgYi9pbmNsdWRlL2xpbnV4L3BndGFibGUuaApAQCAtMTQyNyw2ICsxNDI3LDE2IEBAIHR5cGVk ZWYgdW5zaWduZWQgaW50IHBndGJsX21vZF9tYXNrOwogI2RlZmluZSBtbV9wbWRfZm9sZGVkKG1t KQlfX2lzX2RlZmluZWQoX19QQUdFVEFCTEVfUE1EX0ZPTERFRCkKICNlbmRpZgogCisjaWZuZGVm IHA0ZF9vZmZzZXRfbG9ja2xlc3MKKyNkZWZpbmUgcDRkX29mZnNldF9sb2NrbGVzcyhwZ2RwLCBw Z2QsIGFkZHJlc3MpIHA0ZF9vZmZzZXQoJnBnZCwgYWRkcmVzcykKKyNlbmRpZgorI2lmbmRlZiBw dWRfb2Zmc2V0X2xvY2tsZXNzCisjZGVmaW5lIHB1ZF9vZmZzZXRfbG9ja2xlc3MocDRkcCwgcDRk LCBhZGRyZXNzKSBwdWRfb2Zmc2V0KCZwNGQsIGFkZHJlc3MpCisjZW5kaWYKKyNpZm5kZWYgcG1k X29mZnNldF9sb2NrbGVzcworI2RlZmluZSBwbWRfb2Zmc2V0X2xvY2tsZXNzKHB1ZHAsIHB1ZCwg YWRkcmVzcykgcG1kX29mZnNldCgmcHVkLCBhZGRyZXNzKQorI2VuZGlmCisKIC8qCiAgKiBwP2Rf bGVhZigpIC0gdHJ1ZSBpZiB0aGlzIGVudHJ5IGlzIGEgZmluYWwgbWFwcGluZyB0byBhIHBoeXNp Y2FsIGFkZHJlc3MuCiAgKiBUaGlzIGRpZmZlcnMgZnJvbSBwP2RfaHVnZSgpIGJ5IHRoZSBmYWN0 IHRoYXQgdGhleSBhcmUgYWx3YXlzIGF2YWlsYWJsZSAoaWYKZGlmZiAtLWdpdCBhL21tL2d1cC5j IGIvbW0vZ3VwLmMKaW5kZXggZTU3MzlhMTk3NGQ1Li41NzhiZjViZDhiZjggMTAwNjQ0Ci0tLSBh L21tL2d1cC5jCisrKyBiL21tL2d1cC5jCkBAIC0yNDg1LDEzICsyNDg1LDEzIEBAIHN0YXRpYyBp bnQgZ3VwX2h1Z2VfcGdkKHBnZF90IG9yaWcsIHBnZF90ICpwZ2RwLCB1bnNpZ25lZCBsb25nIGFk ZHIsCiAJcmV0dXJuIDE7CiB9CiAKLXN0YXRpYyBpbnQgZ3VwX3BtZF9yYW5nZShwdWRfdCBwdWQs IHVuc2lnbmVkIGxvbmcgYWRkciwgdW5zaWduZWQgbG9uZyBlbmQsCitzdGF0aWMgaW50IGd1cF9w bWRfcmFuZ2UocHVkX3QgKnB1ZHAsIHB1ZF90IHB1ZCwgdW5zaWduZWQgbG9uZyBhZGRyLCB1bnNp Z25lZCBsb25nIGVuZCwKIAkJdW5zaWduZWQgaW50IGZsYWdzLCBzdHJ1Y3QgcGFnZSAqKnBhZ2Vz LCBpbnQgKm5yKQogewogCXVuc2lnbmVkIGxvbmcgbmV4dDsKIAlwbWRfdCAqcG1kcDsKIAotCXBt ZHAgPSBwbWRfb2Zmc2V0KCZwdWQsIGFkZHIpOworCXBtZHAgPSBwbWRfb2Zmc2V0X2xvY2tsZXNz KHB1ZHAsIHB1ZCwgYWRkcik7CiAJZG8gewogCQlwbWRfdCBwbWQgPSBSRUFEX09OQ0UoKnBtZHAp OwogCkBAIC0yNTI4LDEzICsyNTI4LDEzIEBAIHN0YXRpYyBpbnQgZ3VwX3BtZF9yYW5nZShwdWRf dCBwdWQsIHVuc2lnbmVkIGxvbmcgYWRkciwgdW5zaWduZWQgbG9uZyBlbmQsCiAJcmV0dXJuIDE7 CiB9CiAKLXN0YXRpYyBpbnQgZ3VwX3B1ZF9yYW5nZShwNGRfdCBwNGQsIHVuc2lnbmVkIGxvbmcg YWRkciwgdW5zaWduZWQgbG9uZyBlbmQsCitzdGF0aWMgaW50IGd1cF9wdWRfcmFuZ2UocDRkX3Qg KnA0ZHAsIHA0ZF90IHA0ZCwgdW5zaWduZWQgbG9uZyBhZGRyLCB1bnNpZ25lZCBsb25nIGVuZCwK IAkJCSB1bnNpZ25lZCBpbnQgZmxhZ3MsIHN0cnVjdCBwYWdlICoqcGFnZXMsIGludCAqbnIpCiB7 CiAJdW5zaWduZWQgbG9uZyBuZXh0OwogCXB1ZF90ICpwdWRwOwogCi0JcHVkcCA9IHB1ZF9vZmZz ZXQoJnA0ZCwgYWRkcik7CisJcHVkcCA9IHB1ZF9vZmZzZXRfbG9ja2xlc3MocDRkcCwgcDRkLCBh ZGRyKTsKIAlkbyB7CiAJCXB1ZF90IHB1ZCA9IFJFQURfT05DRSgqcHVkcCk7CiAKQEAgLTI1NDks MjAgKzI1NDksMjAgQEAgc3RhdGljIGludCBndXBfcHVkX3JhbmdlKHA0ZF90IHA0ZCwgdW5zaWdu ZWQgbG9uZyBhZGRyLCB1bnNpZ25lZCBsb25nIGVuZCwKIAkJCWlmICghZ3VwX2h1Z2VfcGQoX19o dWdlcGQocHVkX3ZhbChwdWQpKSwgYWRkciwKIAkJCQkJIFBVRF9TSElGVCwgbmV4dCwgZmxhZ3Ms IHBhZ2VzLCBucikpCiAJCQkJcmV0dXJuIDA7Ci0JCX0gZWxzZSBpZiAoIWd1cF9wbWRfcmFuZ2Uo cHVkLCBhZGRyLCBuZXh0LCBmbGFncywgcGFnZXMsIG5yKSkKKwkJfSBlbHNlIGlmICghZ3VwX3Bt ZF9yYW5nZShwdWRwLCBwdWQsIGFkZHIsIG5leHQsIGZsYWdzLCBwYWdlcywgbnIpKQogCQkJcmV0 dXJuIDA7CiAJfSB3aGlsZSAocHVkcCsrLCBhZGRyID0gbmV4dCwgYWRkciAhPSBlbmQpOwogCiAJ cmV0dXJuIDE7CiB9CiAKLXN0YXRpYyBpbnQgZ3VwX3A0ZF9yYW5nZShwZ2RfdCBwZ2QsIHVuc2ln bmVkIGxvbmcgYWRkciwgdW5zaWduZWQgbG9uZyBlbmQsCitzdGF0aWMgaW50IGd1cF9wNGRfcmFu Z2UocGdkX3QgKnBnZHAsIHBnZF90IHBnZCwgdW5zaWduZWQgbG9uZyBhZGRyLCB1bnNpZ25lZCBs b25nIGVuZCwKIAkJCSB1bnNpZ25lZCBpbnQgZmxhZ3MsIHN0cnVjdCBwYWdlICoqcGFnZXMsIGlu dCAqbnIpCiB7CiAJdW5zaWduZWQgbG9uZyBuZXh0OwogCXA0ZF90ICpwNGRwOwogCi0JcDRkcCA9 IHA0ZF9vZmZzZXQoJnBnZCwgYWRkcik7CisJcDRkcCA9IHA0ZF9vZmZzZXRfbG9ja2xlc3MocGdk cCwgcGdkLCBhZGRyKTsKIAlkbyB7CiAJCXA0ZF90IHA0ZCA9IFJFQURfT05DRSgqcDRkcCk7CiAK QEAgLTI1NzQsNyArMjU3NCw3IEBAIHN0YXRpYyBpbnQgZ3VwX3A0ZF9yYW5nZShwZ2RfdCBwZ2Qs IHVuc2lnbmVkIGxvbmcgYWRkciwgdW5zaWduZWQgbG9uZyBlbmQsCiAJCQlpZiAoIWd1cF9odWdl X3BkKF9faHVnZXBkKHA0ZF92YWwocDRkKSksIGFkZHIsCiAJCQkJCSBQNERfU0hJRlQsIG5leHQs IGZsYWdzLCBwYWdlcywgbnIpKQogCQkJCXJldHVybiAwOwotCQl9IGVsc2UgaWYgKCFndXBfcHVk X3JhbmdlKHA0ZCwgYWRkciwgbmV4dCwgZmxhZ3MsIHBhZ2VzLCBucikpCisJCX0gZWxzZSBpZiAo IWd1cF9wdWRfcmFuZ2UocDRkcCwgcDRkLCBhZGRyLCBuZXh0LCBmbGFncywgcGFnZXMsIG5yKSkK IAkJCXJldHVybiAwOwogCX0gd2hpbGUgKHA0ZHArKywgYWRkciA9IG5leHQsIGFkZHIgIT0gZW5k KTsKIApAQCAtMjYwMiw3ICsyNjAyLDcgQEAgc3RhdGljIHZvaWQgZ3VwX3BnZF9yYW5nZSh1bnNp Z25lZCBsb25nIGFkZHIsIHVuc2lnbmVkIGxvbmcgZW5kLAogCQkJaWYgKCFndXBfaHVnZV9wZChf X2h1Z2VwZChwZ2RfdmFsKHBnZCkpLCBhZGRyLAogCQkJCQkgUEdESVJfU0hJRlQsIG5leHQsIGZs YWdzLCBwYWdlcywgbnIpKQogCQkJCXJldHVybjsKLQkJfSBlbHNlIGlmICghZ3VwX3A0ZF9yYW5n ZShwZ2QsIGFkZHIsIG5leHQsIGZsYWdzLCBwYWdlcywgbnIpKQorCQl9IGVsc2UgaWYgKCFndXBf cDRkX3JhbmdlKHBnZHAsIHBnZCwgYWRkciwgbmV4dCwgZmxhZ3MsIHBhZ2VzLCBucikpCiAJCQly ZXR1cm47CiAJfSB3aGlsZSAocGdkcCsrLCBhZGRyID0gbmV4dCwgYWRkciAhPSBlbmQpOwogfQot LSAK4qO/4qO/4qO/4qO/4qKL4qGA4qOA4qC54qO/4qO/4qO/4qO/CuKjv+Kjv+Kjv+Kjv+KgoOKj tuKhpuKggOKjv+Kjv+Kjv+Kjvwrio7/io7/io7/ioI/io7Tio67io7Tio6fioIjior/io7/io78K 4qO/4qO/4qGP4qKw4qO/4qCW4qOg4qO/4qGG4qCI4qO/4qO/CuKjv+Kim+KjteKjhOKgmeKjtuKj tuKhn+KjheKjoOKgueKjvwrio7/io5zio5vioLvioo7io4nio4nio4DioL/io6vio7Xio78KCl9f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCmxpbnV4LXVtIG1h aWxpbmcgbGlzdApsaW51eC11bUBsaXN0cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0cy5pbmZy YWRlYWQub3JnL21haWxtYW4vbGlzdGluZm8vbGludXgtdW0K From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95EA6C433E2 for ; Fri, 11 Sep 2020 23:12:05 +0000 (UTC) Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2C657221F0 for ; Fri, 11 Sep 2020 23:12:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="PAoqGBmI" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2C657221F0 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4BpBNK2YX5zDqx5 for ; Sat, 12 Sep 2020 09:12:01 +1000 (AEST) Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=gor@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=PAoqGBmI; dkim-atps=neutral Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Bp4tR2nTZzDqv3 for ; Sat, 12 Sep 2020 05:04:11 +1000 (AEST) Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 08BJ1nqJ070282; Fri, 11 Sep 2020 15:03:16 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=date : from : to : cc : subject : message-id : references : mime-version : content-type : content-transfer-encoding : in-reply-to; s=pp1; bh=BA+waGK444WFdAdGsxtmH4MVyQmh9mHMY6P6ku0v4Ks=; b=PAoqGBmI1WsEu/qH5mX1NCJjx4VjyBJDOUw68p7GuYHEFTEDLVyHklhTy+D05a9nhIEh S3OPtLLBBYAxRCIWYIGMEgVuSzV/wiY0aRinuDzcFKAICBrC4V6KZGYgqhBPlJ/HtW90 2sVdth6sFOm8qAfahbo8bWOSGUQUEITM97sVy3F3h16HR0GwbY8iTIlfBFmJ+neomaWW ttvdIJBuVK0LzRf3TnGmPJDMF9t2Br4ehU8orVA+6UbKHdmO/NDqhkIh6K8Prm4HrnX/ X/r9PwkYXTaBXZJhLWPSWh77gi2NbCByHKwD02OEDWOrr1Md8dXOnOojvEUCnbXgmBbM gw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 33geac150s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 11 Sep 2020 15:03:16 -0400 Received: from m0098394.ppops.net (m0098394.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 08BJ2G5J073420; Fri, 11 Sep 2020 15:03:15 -0400 Received: from ppma04ams.nl.ibm.com (63.31.33a9.ip4.static.sl-reverse.com [169.51.49.99]) by mx0a-001b2d01.pphosted.com with ESMTP id 33geac14yv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 11 Sep 2020 15:03:15 -0400 Received: from pps.filterd (ppma04ams.nl.ibm.com [127.0.0.1]) by ppma04ams.nl.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 08BIvA0f002964; Fri, 11 Sep 2020 19:03:12 GMT Received: from b06cxnps4074.portsmouth.uk.ibm.com (d06relay11.portsmouth.uk.ibm.com [9.149.109.196]) by ppma04ams.nl.ibm.com with ESMTP id 33c2a8fk1b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 11 Sep 2020 19:03:12 +0000 Received: from b06wcsmtp001.portsmouth.uk.ibm.com (b06wcsmtp001.portsmouth.uk.ibm.com [9.149.105.160]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 08BJ39xe22675840 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 11 Sep 2020 19:03:10 GMT Received: from b06wcsmtp001.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B6F01A405F; Fri, 11 Sep 2020 19:03:09 +0000 (GMT) Received: from b06wcsmtp001.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 02C6DA405B; Fri, 11 Sep 2020 19:03:08 +0000 (GMT) Received: from localhost (unknown [9.145.43.16]) by b06wcsmtp001.portsmouth.uk.ibm.com (Postfix) with ESMTPS; Fri, 11 Sep 2020 19:03:07 +0000 (GMT) Date: Fri, 11 Sep 2020 21:03:06 +0200 From: Vasily Gorbik To: Jason Gunthorpe , John Hubbard , Linus Torvalds Subject: [PATCH] mm/gup: fix gup_fast with dynamic page table folding Message-ID: References: <20200911070939.GB1362448@hirez.programming.kicks-ass.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20200911070939.GB1362448@hirez.programming.kicks-ass.net> X-Patchwork-Bot: notify X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235, 18.0.687 definitions=2020-09-11_09:2020-09-10, 2020-09-11 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 impostorscore=0 lowpriorityscore=0 priorityscore=1501 malwarescore=0 mlxlogscore=999 phishscore=0 bulkscore=0 suspectscore=0 spamscore=0 adultscore=0 clxscore=1011 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2009110148 X-Mailman-Approved-At: Sat, 12 Sep 2020 09:10:03 +1000 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Zijlstra , Dave Hansen , linux-mm , Paul Mackerras , linux-sparc , Alexander Gordeev , Claudio Imbrenda , Will Deacon , linux-arch , linux-s390 , Richard Weinberger , linux-x86 , Russell King , Christian Borntraeger , Ingo Molnar , Catalin Marinas , Andrey Ryabinin , Gerald Schaefer , Heiko Carstens , Arnd Bergmann , Jeff Dike , linux-um , Borislav Petkov , Andy Lutomirski , Thomas Gleixner , linux-arm , Dave Hansen , LKML , Andrew Morton , linux-power , Mike Rapoport Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" Currently to make sure that every page table entry is read just once gup_fast walks perform READ_ONCE and pass pXd value down to the next gup_pXd_range function by value e.g.: static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) ... pudp = pud_offset(&p4d, addr); This function passes a reference on that local value copy to pXd_offset, and might get the very same pointer in return. This happens when the level is folded (on most arches), and that pointer should not be iterated. On s390 due to the fact that each task might have different 5,4 or 3-level address translation and hence different levels folded the logic is more complex and non-iteratable pointer to a local copy leads to severe problems. Here is an example of what happens with gup_fast on s390, for a task with 3-levels paging, crossing a 2 GB pud boundary: // addr = 0x1007ffff000, end = 0x10080001000 static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { unsigned long next; pud_t *pudp; // pud_offset returns &p4d itself (a pointer to a value on stack) pudp = pud_offset(&p4d, addr); do { // on second iteratation reading "random" stack value pud_t pud = READ_ONCE(*pudp); // next = 0x10080000000, due to PUD_SIZE/MASK != PGDIR_SIZE/MASK on s390 next = pud_addr_end(addr, end); ... } while (pudp++, addr = next, addr != end); // pudp++ iterating over stack return 1; } This happens since s390 moved to common gup code with commit d1874a0c2805 ("s390/mm: make the pxd_offset functions more robust") and commit 1a42010cdc26 ("s390/mm: convert to the generic get_user_pages_fast code"). s390 tried to mimic static level folding by changing pXd_offset primitives to always calculate top level page table offset in pgd_offset and just return the value passed when pXd_offset has to act as folded. What is crucial for gup_fast and what has been overlooked is that PxD_SIZE/MASK and thus pXd_addr_end should also change correspondingly. And the latter is not possible with dynamic folding. To fix the issue in addition to pXd values pass original pXdp pointers down to gup_pXd_range functions. And introduce pXd_offset_lockless helpers, which take an additional pXd entry value parameter. This has already been discussed in https://lkml.kernel.org/r/20190418100218.0a4afd51@mschwideX1 Cc: # 5.2+ Fixes: 1a42010cdc26 ("s390/mm: convert to the generic get_user_pages_fast code") Reviewed-by: Gerald Schaefer Reviewed-by: Alexander Gordeev Signed-off-by: Vasily Gorbik --- arch/s390/include/asm/pgtable.h | 42 +++++++++++++++++++++++---------- include/linux/pgtable.h | 10 ++++++++ mm/gup.c | 18 +++++++------- 3 files changed, 49 insertions(+), 21 deletions(-) diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h index 7eb01a5459cd..b55561cc8786 100644 --- a/arch/s390/include/asm/pgtable.h +++ b/arch/s390/include/asm/pgtable.h @@ -1260,26 +1260,44 @@ static inline pgd_t *pgd_offset_raw(pgd_t *pgd, unsigned long address) #define pgd_offset(mm, address) pgd_offset_raw(READ_ONCE((mm)->pgd), address) -static inline p4d_t *p4d_offset(pgd_t *pgd, unsigned long address) +static inline p4d_t *p4d_offset_lockless(pgd_t *pgdp, pgd_t pgd, unsigned long address) { - if ((pgd_val(*pgd) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R1) - return (p4d_t *) pgd_deref(*pgd) + p4d_index(address); - return (p4d_t *) pgd; + if ((pgd_val(pgd) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R1) + return (p4d_t *) pgd_deref(pgd) + p4d_index(address); + return (p4d_t *) pgdp; } +#define p4d_offset_lockless p4d_offset_lockless -static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address) +static inline p4d_t *p4d_offset(pgd_t *pgdp, unsigned long address) { - if ((p4d_val(*p4d) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R2) - return (pud_t *) p4d_deref(*p4d) + pud_index(address); - return (pud_t *) p4d; + return p4d_offset_lockless(pgdp, *pgdp, address); +} + +static inline pud_t *pud_offset_lockless(p4d_t *p4dp, p4d_t p4d, unsigned long address) +{ + if ((p4d_val(p4d) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R2) + return (pud_t *) p4d_deref(p4d) + pud_index(address); + return (pud_t *) p4dp; +} +#define pud_offset_lockless pud_offset_lockless + +static inline pud_t *pud_offset(p4d_t *p4dp, unsigned long address) +{ + return pud_offset_lockless(p4dp, *p4dp, address); } #define pud_offset pud_offset -static inline pmd_t *pmd_offset(pud_t *pud, unsigned long address) +static inline pmd_t *pmd_offset_lockless(pud_t *pudp, pud_t pud, unsigned long address) +{ + if ((pud_val(pud) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R3) + return (pmd_t *) pud_deref(pud) + pmd_index(address); + return (pmd_t *) pudp; +} +#define pmd_offset_lockless pmd_offset_lockless + +static inline pmd_t *pmd_offset(pud_t *pudp, unsigned long address) { - if ((pud_val(*pud) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R3) - return (pmd_t *) pud_deref(*pud) + pmd_index(address); - return (pmd_t *) pud; + return pmd_offset_lockless(pudp, *pudp, address); } #define pmd_offset pmd_offset diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index e8cbc2e795d5..e899d3506671 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1427,6 +1427,16 @@ typedef unsigned int pgtbl_mod_mask; #define mm_pmd_folded(mm) __is_defined(__PAGETABLE_PMD_FOLDED) #endif +#ifndef p4d_offset_lockless +#define p4d_offset_lockless(pgdp, pgd, address) p4d_offset(&pgd, address) +#endif +#ifndef pud_offset_lockless +#define pud_offset_lockless(p4dp, p4d, address) pud_offset(&p4d, address) +#endif +#ifndef pmd_offset_lockless +#define pmd_offset_lockless(pudp, pud, address) pmd_offset(&pud, address) +#endif + /* * p?d_leaf() - true if this entry is a final mapping to a physical address. * This differs from p?d_huge() by the fact that they are always available (if diff --git a/mm/gup.c b/mm/gup.c index e5739a1974d5..578bf5bd8bf8 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2485,13 +2485,13 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, return 1; } -static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end, +static int gup_pmd_range(pud_t *pudp, pud_t pud, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { unsigned long next; pmd_t *pmdp; - pmdp = pmd_offset(&pud, addr); + pmdp = pmd_offset_lockless(pudp, pud, addr); do { pmd_t pmd = READ_ONCE(*pmdp); @@ -2528,13 +2528,13 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end, return 1; } -static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, +static int gup_pud_range(p4d_t *p4dp, p4d_t p4d, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { unsigned long next; pud_t *pudp; - pudp = pud_offset(&p4d, addr); + pudp = pud_offset_lockless(p4dp, p4d, addr); do { pud_t pud = READ_ONCE(*pudp); @@ -2549,20 +2549,20 @@ static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, if (!gup_huge_pd(__hugepd(pud_val(pud)), addr, PUD_SHIFT, next, flags, pages, nr)) return 0; - } else if (!gup_pmd_range(pud, addr, next, flags, pages, nr)) + } else if (!gup_pmd_range(pudp, pud, addr, next, flags, pages, nr)) return 0; } while (pudp++, addr = next, addr != end); return 1; } -static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end, +static int gup_p4d_range(pgd_t *pgdp, pgd_t pgd, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { unsigned long next; p4d_t *p4dp; - p4dp = p4d_offset(&pgd, addr); + p4dp = p4d_offset_lockless(pgdp, pgd, addr); do { p4d_t p4d = READ_ONCE(*p4dp); @@ -2574,7 +2574,7 @@ static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end, if (!gup_huge_pd(__hugepd(p4d_val(p4d)), addr, P4D_SHIFT, next, flags, pages, nr)) return 0; - } else if (!gup_pud_range(p4d, addr, next, flags, pages, nr)) + } else if (!gup_pud_range(p4dp, p4d, addr, next, flags, pages, nr)) return 0; } while (p4dp++, addr = next, addr != end); @@ -2602,7 +2602,7 @@ static void gup_pgd_range(unsigned long addr, unsigned long end, if (!gup_huge_pd(__hugepd(pgd_val(pgd)), addr, PGDIR_SHIFT, next, flags, pages, nr)) return; - } else if (!gup_p4d_range(pgd, addr, next, flags, pages, nr)) + } else if (!gup_p4d_range(pgdp, pgd, addr, next, flags, pages, nr)) return; } while (pgdp++, addr = next, addr != end); } -- ⣿⣿⣿⣿⢋⡀⣀⠹⣿⣿⣿⣿ ⣿⣿⣿⣿⠠⣶⡦⠀⣿⣿⣿⣿ ⣿⣿⣿⠏⣴⣮⣴⣧⠈⢿⣿⣿ ⣿⣿⡏⢰⣿⠖⣠⣿⡆⠈⣿⣿ ⣿⢛⣵⣄⠙⣶⣶⡟⣅⣠⠹⣿ ⣿⣜⣛⠻⢎⣉⣉⣀⠿⣫⣵⣿ From mboxrd@z Thu Jan 1 00:00:00 1970 From: Vasily Gorbik Date: Fri, 11 Sep 2020 19:03:06 +0000 Subject: [PATCH] mm/gup: fix gup_fast with dynamic page table folding Message-Id: List-Id: References: <20200911070939.GB1362448@hirez.programming.kicks-ass.net> In-Reply-To: <20200911070939.GB1362448@hirez.programming.kicks-ass.net> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit To: Jason Gunthorpe , John Hubbard , Linus Torvalds Cc: Gerald Schaefer , Alexander Gordeev , Peter Zijlstra , Dave Hansen , LKML , linux-mm , linux-arch , Andrew Morton , Russell King , Mike Rapoport , Catalin Marinas , Will Deacon , Michael Ellerman , Benjamin Herrenschmidt , Paul Mackerras , Jeff Dike , Richard Weinberger , Dave Hansen , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Arnd Bergmann , Andrey Ryabinin , linux-x86 , linux-arm , linux-power , linux-sparc , linux-um , linux-s390 , Heiko Carstens , Christian Borntraeger , Claudio Imbrenda Currently to make sure that every page table entry is read just once gup_fast walks perform READ_ONCE and pass pXd value down to the next gup_pXd_range function by value e.g.: static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) ... pudp = pud_offset(&p4d, addr); This function passes a reference on that local value copy to pXd_offset, and might get the very same pointer in return. This happens when the level is folded (on most arches), and that pointer should not be iterated. On s390 due to the fact that each task might have different 5,4 or 3-level address translation and hence different levels folded the logic is more complex and non-iteratable pointer to a local copy leads to severe problems. Here is an example of what happens with gup_fast on s390, for a task with 3-levels paging, crossing a 2 GB pud boundary: // addr = 0x1007ffff000, end = 0x10080001000 static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { unsigned long next; pud_t *pudp; // pud_offset returns &p4d itself (a pointer to a value on stack) pudp = pud_offset(&p4d, addr); do { // on second iteratation reading "random" stack value pud_t pud = READ_ONCE(*pudp); // next = 0x10080000000, due to PUD_SIZE/MASK != PGDIR_SIZE/MASK on s390 next = pud_addr_end(addr, end); ... } while (pudp++, addr = next, addr != end); // pudp++ iterating over stack return 1; } This happens since s390 moved to common gup code with commit d1874a0c2805 ("s390/mm: make the pxd_offset functions more robust") and commit 1a42010cdc26 ("s390/mm: convert to the generic get_user_pages_fast code"). s390 tried to mimic static level folding by changing pXd_offset primitives to always calculate top level page table offset in pgd_offset and just return the value passed when pXd_offset has to act as folded. What is crucial for gup_fast and what has been overlooked is that PxD_SIZE/MASK and thus pXd_addr_end should also change correspondingly. And the latter is not possible with dynamic folding. To fix the issue in addition to pXd values pass original pXdp pointers down to gup_pXd_range functions. And introduce pXd_offset_lockless helpers, which take an additional pXd entry value parameter. This has already been discussed in https://lkml.kernel.org/r/20190418100218.0a4afd51@mschwideX1 Cc: # 5.2+ Fixes: 1a42010cdc26 ("s390/mm: convert to the generic get_user_pages_fast code") Reviewed-by: Gerald Schaefer Reviewed-by: Alexander Gordeev Signed-off-by: Vasily Gorbik --- arch/s390/include/asm/pgtable.h | 42 +++++++++++++++++++++++---------- include/linux/pgtable.h | 10 ++++++++ mm/gup.c | 18 +++++++------- 3 files changed, 49 insertions(+), 21 deletions(-) diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h index 7eb01a5459cd..b55561cc8786 100644 --- a/arch/s390/include/asm/pgtable.h +++ b/arch/s390/include/asm/pgtable.h @@ -1260,26 +1260,44 @@ static inline pgd_t *pgd_offset_raw(pgd_t *pgd, unsigned long address) #define pgd_offset(mm, address) pgd_offset_raw(READ_ONCE((mm)->pgd), address) -static inline p4d_t *p4d_offset(pgd_t *pgd, unsigned long address) +static inline p4d_t *p4d_offset_lockless(pgd_t *pgdp, pgd_t pgd, unsigned long address) { - if ((pgd_val(*pgd) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R1) - return (p4d_t *) pgd_deref(*pgd) + p4d_index(address); - return (p4d_t *) pgd; + if ((pgd_val(pgd) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R1) + return (p4d_t *) pgd_deref(pgd) + p4d_index(address); + return (p4d_t *) pgdp; } +#define p4d_offset_lockless p4d_offset_lockless -static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address) +static inline p4d_t *p4d_offset(pgd_t *pgdp, unsigned long address) { - if ((p4d_val(*p4d) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R2) - return (pud_t *) p4d_deref(*p4d) + pud_index(address); - return (pud_t *) p4d; + return p4d_offset_lockless(pgdp, *pgdp, address); +} + +static inline pud_t *pud_offset_lockless(p4d_t *p4dp, p4d_t p4d, unsigned long address) +{ + if ((p4d_val(p4d) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R2) + return (pud_t *) p4d_deref(p4d) + pud_index(address); + return (pud_t *) p4dp; +} +#define pud_offset_lockless pud_offset_lockless + +static inline pud_t *pud_offset(p4d_t *p4dp, unsigned long address) +{ + return pud_offset_lockless(p4dp, *p4dp, address); } #define pud_offset pud_offset -static inline pmd_t *pmd_offset(pud_t *pud, unsigned long address) +static inline pmd_t *pmd_offset_lockless(pud_t *pudp, pud_t pud, unsigned long address) +{ + if ((pud_val(pud) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R3) + return (pmd_t *) pud_deref(pud) + pmd_index(address); + return (pmd_t *) pudp; +} +#define pmd_offset_lockless pmd_offset_lockless + +static inline pmd_t *pmd_offset(pud_t *pudp, unsigned long address) { - if ((pud_val(*pud) & _REGION_ENTRY_TYPE_MASK) >= _REGION_ENTRY_TYPE_R3) - return (pmd_t *) pud_deref(*pud) + pmd_index(address); - return (pmd_t *) pud; + return pmd_offset_lockless(pudp, *pudp, address); } #define pmd_offset pmd_offset diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index e8cbc2e795d5..e899d3506671 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1427,6 +1427,16 @@ typedef unsigned int pgtbl_mod_mask; #define mm_pmd_folded(mm) __is_defined(__PAGETABLE_PMD_FOLDED) #endif +#ifndef p4d_offset_lockless +#define p4d_offset_lockless(pgdp, pgd, address) p4d_offset(&pgd, address) +#endif +#ifndef pud_offset_lockless +#define pud_offset_lockless(p4dp, p4d, address) pud_offset(&p4d, address) +#endif +#ifndef pmd_offset_lockless +#define pmd_offset_lockless(pudp, pud, address) pmd_offset(&pud, address) +#endif + /* * p?d_leaf() - true if this entry is a final mapping to a physical address. * This differs from p?d_huge() by the fact that they are always available (if diff --git a/mm/gup.c b/mm/gup.c index e5739a1974d5..578bf5bd8bf8 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2485,13 +2485,13 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, return 1; } -static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end, +static int gup_pmd_range(pud_t *pudp, pud_t pud, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { unsigned long next; pmd_t *pmdp; - pmdp = pmd_offset(&pud, addr); + pmdp = pmd_offset_lockless(pudp, pud, addr); do { pmd_t pmd = READ_ONCE(*pmdp); @@ -2528,13 +2528,13 @@ static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end, return 1; } -static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, +static int gup_pud_range(p4d_t *p4dp, p4d_t p4d, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { unsigned long next; pud_t *pudp; - pudp = pud_offset(&p4d, addr); + pudp = pud_offset_lockless(p4dp, p4d, addr); do { pud_t pud = READ_ONCE(*pudp); @@ -2549,20 +2549,20 @@ static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, if (!gup_huge_pd(__hugepd(pud_val(pud)), addr, PUD_SHIFT, next, flags, pages, nr)) return 0; - } else if (!gup_pmd_range(pud, addr, next, flags, pages, nr)) + } else if (!gup_pmd_range(pudp, pud, addr, next, flags, pages, nr)) return 0; } while (pudp++, addr = next, addr != end); return 1; } -static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end, +static int gup_p4d_range(pgd_t *pgdp, pgd_t pgd, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { unsigned long next; p4d_t *p4dp; - p4dp = p4d_offset(&pgd, addr); + p4dp = p4d_offset_lockless(pgdp, pgd, addr); do { p4d_t p4d = READ_ONCE(*p4dp); @@ -2574,7 +2574,7 @@ static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end, if (!gup_huge_pd(__hugepd(p4d_val(p4d)), addr, P4D_SHIFT, next, flags, pages, nr)) return 0; - } else if (!gup_pud_range(p4d, addr, next, flags, pages, nr)) + } else if (!gup_pud_range(p4dp, p4d, addr, next, flags, pages, nr)) return 0; } while (p4dp++, addr = next, addr != end); @@ -2602,7 +2602,7 @@ static void gup_pgd_range(unsigned long addr, unsigned long end, if (!gup_huge_pd(__hugepd(pgd_val(pgd)), addr, PGDIR_SHIFT, next, flags, pages, nr)) return; - } else if (!gup_p4d_range(pgd, addr, next, flags, pages, nr)) + } else if (!gup_p4d_range(pgdp, pgd, addr, next, flags, pages, nr)) return; } while (pgdp++, addr = next, addr != end); } -- ⣿⣿⣿⣿⢋⡀⣀⠹⣿⣿⣿⣿ ⣿⣿⣿⣿⠠⣶⡦⠀⣿⣿⣿⣿ ⣿⣿⣿⠏⣴⣮⣴⣧⠈⢿⣿⣿ ⣿⣿⡏⢰⣿⠖⣠⣿⡆⠈⣿⣿ ⣿⢛⣵⣄⠙⣶⣶⡟⣅⣠⠹⣿ ⣿⣜⣛⠻⢎⣉⣉⣀⠿⣫⣵⣿ From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ECB7EC433E2 for ; Fri, 11 Sep 2020 19:05:39 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8A6CB206E9 for ; Fri, 11 Sep 2020 19:05:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="2NjjlRYf"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="PAoqGBmI" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8A6CB206E9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=g5IplnScYvf32qWFsVYwVtBcZINWfRXXhcEfu67Hv0U=; b=2NjjlRYfV4riZy80xmFY/3t68 dgZwQD9HqugMX4M5yTeQOUAX4WUvERccl9dC5cnWfewtYPD3naSazyeQ5ShwiHyI8Wk1yDAqEOgzZ iF9YYqxkvgRwhn3yiEG4SZDU6DXVR5CsWPn6g5luAwHSisTfA9YlGlyOYJF3k9q4Bp8OJT+as162e QymkJBpQczBYu+b0lZittOSka7diDG6foTR2PMBFtC2nb/N53E10hKQ3jh+3tvBrg2atfj2Fzh0wC m0mC86OMD9CqCVIuNUQRR8LnHluP5ACibdsATFhYDxfpmjl8IHGL3Z6YfdizUJUolrHkRnFGacsvz Awi1oRmMg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kGoLD-0006iV-E5; Fri, 11 Sep 2020 19:04:11 +0000 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kGoLA-0006i3-Mi; Fri, 11 Sep 2020 19:04:09 +0000 Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 08BJ1nqJ070282; Fri, 11 Sep 2020 15:03:16 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=date : from : to : cc : subject : message-id : references : mime-version : content-type : content-transfer-encoding : in-reply-to; s=pp1; bh=BA+waGK444WFdAdGsxtmH4MVyQmh9mHMY6P6ku0v4Ks=; b=PAoqGBmI1WsEu/qH5mX1NCJjx4VjyBJDOUw68p7GuYHEFTEDLVyHklhTy+D05a9nhIEh S3OPtLLBBYAxRCIWYIGMEgVuSzV/wiY0aRinuDzcFKAICBrC4V6KZGYgqhBPlJ/HtW90 2sVdth6sFOm8qAfahbo8bWOSGUQUEITM97sVy3F3h16HR0GwbY8iTIlfBFmJ+neomaWW ttvdIJBuVK0LzRf3TnGmPJDMF9t2Br4ehU8orVA+6UbKHdmO/NDqhkIh6K8Prm4HrnX/ X/r9PwkYXTaBXZJhLWPSWh77gi2NbCByHKwD02OEDWOrr1Md8dXOnOojvEUCnbXgmBbM gw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 33geac150s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 11 Sep 2020 15:03:16 -0400 Received: from m0098394.ppops.net (m0098394.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 08BJ2G5J073420; Fri, 11 Sep 2020 15:03:15 -0400 Received: from ppma04ams.nl.ibm.com (63.31.33a9.ip4.static.sl-reverse.com [169.51.49.99]) by mx0a-001b2d01.pphosted.com with ESMTP id 33geac14yv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 11 Sep 2020 15:03:15 -0400 Received: from pps.filterd (ppma04ams.nl.ibm.com [127.0.0.1]) by ppma04ams.nl.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 08BIvA0f002964; Fri, 11 Sep 2020 19:03:12 GMT Received: from b06cxnps4074.portsmouth.uk.ibm.com (d06relay11.portsmouth.uk.ibm.com [9.149.109.196]) by ppma04ams.nl.ibm.com with ESMTP id 33c2a8fk1b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 11 Sep 2020 19:03:12 +0000 Received: from b06wcsmtp001.portsmouth.uk.ibm.com (b06wcsmtp001.portsmouth.uk.ibm.com [9.149.105.160]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 08BJ39xe22675840 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 11 Sep 2020 19:03:10 GMT Received: from b06wcsmtp001.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B6F01A405F; Fri, 11 Sep 2020 19:03:09 +0000 (GMT) Received: from b06wcsmtp001.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 02C6DA405B; Fri, 11 Sep 2020 19:03:08 +0000 (GMT) Received: from localhost (unknown [9.145.43.16]) by b06wcsmtp001.portsmouth.uk.ibm.com (Postfix) with ESMTPS; Fri, 11 Sep 2020 19:03:07 +0000 (GMT) Date: Fri, 11 Sep 2020 21:03:06 +0200 From: Vasily Gorbik To: Jason Gunthorpe , John Hubbard , Linus Torvalds Subject: [PATCH] mm/gup: fix gup_fast with dynamic page table folding Message-ID: References: <20200911070939.GB1362448@hirez.programming.kicks-ass.net> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20200911070939.GB1362448@hirez.programming.kicks-ass.net> X-Patchwork-Bot: notify X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235, 18.0.687 definitions=2020-09-11_09:2020-09-10, 2020-09-11 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 impostorscore=0 lowpriorityscore=0 priorityscore=1501 malwarescore=0 mlxlogscore=999 phishscore=0 bulkscore=0 suspectscore=0 spamscore=0 adultscore=0 clxscore=1011 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2009110148 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200911_150408_899949_F5DA7E66 X-CRM114-Status: GOOD ( 29.85 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Zijlstra , Benjamin Herrenschmidt , Dave Hansen , linux-mm , Paul Mackerras , linux-sparc , Alexander Gordeev , Claudio Imbrenda , Will Deacon , linux-arch , linux-s390 , Richard Weinberger , linux-x86 , Russell King , Christian Borntraeger , Ingo Molnar , Catalin Marinas , Andrey Ryabinin , Gerald Schaefer , Heiko Carstens , Arnd Bergmann , Jeff Dike , linux-um , Borislav Petkov , Andy Lutomirski , Thomas Gleixner , linux-arm , Dave Hansen , LKML , Michael Ellerman , Andrew Morton , linux-power , Mike Rapoport Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Q3VycmVudGx5IHRvIG1ha2Ugc3VyZSB0aGF0IGV2ZXJ5IHBhZ2UgdGFibGUgZW50cnkgaXMgcmVh ZCBqdXN0IG9uY2UKZ3VwX2Zhc3Qgd2Fsa3MgcGVyZm9ybSBSRUFEX09OQ0UgYW5kIHBhc3MgcFhk IHZhbHVlIGRvd24gdG8gdGhlIG5leHQKZ3VwX3BYZF9yYW5nZSBmdW5jdGlvbiBieSB2YWx1ZSBl LmcuOgoKc3RhdGljIGludCBndXBfcHVkX3JhbmdlKHA0ZF90IHA0ZCwgdW5zaWduZWQgbG9uZyBh ZGRyLCB1bnNpZ25lZCBsb25nIGVuZCwKICAgICAgICAgICAgICAgICAgICAgICAgIHVuc2lnbmVk IGludCBmbGFncywgc3RydWN0IHBhZ2UgKipwYWdlcywgaW50ICpucikKLi4uCiAgICAgICAgcHVk cCA9IHB1ZF9vZmZzZXQoJnA0ZCwgYWRkcik7CgpUaGlzIGZ1bmN0aW9uIHBhc3NlcyBhIHJlZmVy ZW5jZSBvbiB0aGF0IGxvY2FsIHZhbHVlIGNvcHkgdG8gcFhkX29mZnNldCwKYW5kIG1pZ2h0IGdl dCB0aGUgdmVyeSBzYW1lIHBvaW50ZXIgaW4gcmV0dXJuLiBUaGlzIGhhcHBlbnMgd2hlbiB0aGUK bGV2ZWwgaXMgZm9sZGVkIChvbiBtb3N0IGFyY2hlcyksIGFuZCB0aGF0IHBvaW50ZXIgc2hvdWxk IG5vdCBiZSBpdGVyYXRlZC4KCk9uIHMzOTAgZHVlIHRvIHRoZSBmYWN0IHRoYXQgZWFjaCB0YXNr IG1pZ2h0IGhhdmUgZGlmZmVyZW50IDUsNCBvcgozLWxldmVsIGFkZHJlc3MgdHJhbnNsYXRpb24g YW5kIGhlbmNlIGRpZmZlcmVudCBsZXZlbHMgZm9sZGVkIHRoZSBsb2dpYwppcyBtb3JlIGNvbXBs ZXggYW5kIG5vbi1pdGVyYXRhYmxlIHBvaW50ZXIgdG8gYSBsb2NhbCBjb3B5IGxlYWRzIHRvCnNl dmVyZSBwcm9ibGVtcy4KCkhlcmUgaXMgYW4gZXhhbXBsZSBvZiB3aGF0IGhhcHBlbnMgd2l0aCBn dXBfZmFzdCBvbiBzMzkwLCBmb3IgYSB0YXNrCndpdGggMy1sZXZlbHMgcGFnaW5nLCBjcm9zc2lu ZyBhIDIgR0IgcHVkIGJvdW5kYXJ5OgoKLy8gYWRkciA9IDB4MTAwN2ZmZmYwMDAsIGVuZCA9IDB4 MTAwODAwMDEwMDAKc3RhdGljIGludCBndXBfcHVkX3JhbmdlKHA0ZF90IHA0ZCwgdW5zaWduZWQg bG9uZyBhZGRyLCB1bnNpZ25lZCBsb25nIGVuZCwKICAgICAgICAgICAgICAgICAgICAgICAgIHVu c2lnbmVkIGludCBmbGFncywgc3RydWN0IHBhZ2UgKipwYWdlcywgaW50ICpucikKewogICAgICAg IHVuc2lnbmVkIGxvbmcgbmV4dDsKICAgICAgICBwdWRfdCAqcHVkcDsKCiAgICAgICAgLy8gcHVk X29mZnNldCByZXR1cm5zICZwNGQgaXRzZWxmIChhIHBvaW50ZXIgdG8gYSB2YWx1ZSBvbiBzdGFj aykKICAgICAgICBwdWRwID0gcHVkX29mZnNldCgmcDRkLCBhZGRyKTsKICAgICAgICBkbyB7CiAg ICAgICAgICAgICAgICAvLyBvbiBzZWNvbmQgaXRlcmF0YXRpb24gcmVhZGluZyAicmFuZG9tIiBz dGFjayB2YWx1ZQogICAgICAgICAgICAgICAgcHVkX3QgcHVkID0gUkVBRF9PTkNFKCpwdWRwKTsK CiAgICAgICAgICAgICAgICAvLyBuZXh0ID0gMHgxMDA4MDAwMDAwMCwgZHVlIHRvIFBVRF9TSVpF L01BU0sgIT0gUEdESVJfU0laRS9NQVNLIG9uIHMzOTAKICAgICAgICAgICAgICAgIG5leHQgPSBw dWRfYWRkcl9lbmQoYWRkciwgZW5kKTsKICAgICAgICAgICAgICAgIC4uLgogICAgICAgIH0gd2hp bGUgKHB1ZHArKywgYWRkciA9IG5leHQsIGFkZHIgIT0gZW5kKTsgLy8gcHVkcCsrIGl0ZXJhdGlu ZyBvdmVyIHN0YWNrCgogICAgICAgIHJldHVybiAxOwp9CgpUaGlzIGhhcHBlbnMgc2luY2UgczM5 MCBtb3ZlZCB0byBjb21tb24gZ3VwIGNvZGUgd2l0aApjb21taXQgZDE4NzRhMGMyODA1ICgiczM5 MC9tbTogbWFrZSB0aGUgcHhkX29mZnNldCBmdW5jdGlvbnMgbW9yZSByb2J1c3QiKQphbmQgY29t bWl0IDFhNDIwMTBjZGMyNiAoInMzOTAvbW06IGNvbnZlcnQgdG8gdGhlIGdlbmVyaWMKZ2V0X3Vz ZXJfcGFnZXNfZmFzdCBjb2RlIikuIHMzOTAgdHJpZWQgdG8gbWltaWMgc3RhdGljIGxldmVsIGZv bGRpbmcgYnkKY2hhbmdpbmcgcFhkX29mZnNldCBwcmltaXRpdmVzIHRvIGFsd2F5cyBjYWxjdWxh dGUgdG9wIGxldmVsIHBhZ2UgdGFibGUKb2Zmc2V0IGluIHBnZF9vZmZzZXQgYW5kIGp1c3QgcmV0 dXJuIHRoZSB2YWx1ZSBwYXNzZWQgd2hlbiBwWGRfb2Zmc2V0CmhhcyB0byBhY3QgYXMgZm9sZGVk LgoKV2hhdCBpcyBjcnVjaWFsIGZvciBndXBfZmFzdCBhbmQgd2hhdCBoYXMgYmVlbiBvdmVybG9v a2VkIGlzCnRoYXQgUHhEX1NJWkUvTUFTSyBhbmQgdGh1cyBwWGRfYWRkcl9lbmQgc2hvdWxkIGFs c28gY2hhbmdlCmNvcnJlc3BvbmRpbmdseS4gQW5kIHRoZSBsYXR0ZXIgaXMgbm90IHBvc3NpYmxl IHdpdGggZHluYW1pYyBmb2xkaW5nLgoKVG8gZml4IHRoZSBpc3N1ZSBpbiBhZGRpdGlvbiB0byBw WGQgdmFsdWVzIHBhc3Mgb3JpZ2luYWwKcFhkcCBwb2ludGVycyBkb3duIHRvIGd1cF9wWGRfcmFu Z2UgZnVuY3Rpb25zLiBBbmQgaW50cm9kdWNlCnBYZF9vZmZzZXRfbG9ja2xlc3MgaGVscGVycywg d2hpY2ggdGFrZSBhbiBhZGRpdGlvbmFsIHBYZAplbnRyeSB2YWx1ZSBwYXJhbWV0ZXIuIFRoaXMg aGFzIGFscmVhZHkgYmVlbiBkaXNjdXNzZWQgaW4KaHR0cHM6Ly9sa21sLmtlcm5lbC5vcmcvci8y MDE5MDQxODEwMDIxOC4wYTRhZmQ1MUBtc2Nod2lkZVgxCgpDYzogPHN0YWJsZUB2Z2VyLmtlcm5l bC5vcmc+ICMgNS4yKwpGaXhlczogMWE0MjAxMGNkYzI2ICgiczM5MC9tbTogY29udmVydCB0byB0 aGUgZ2VuZXJpYyBnZXRfdXNlcl9wYWdlc19mYXN0IGNvZGUiKQpSZXZpZXdlZC1ieTogR2VyYWxk IFNjaGFlZmVyIDxnZXJhbGQuc2NoYWVmZXJAbGludXguaWJtLmNvbT4KUmV2aWV3ZWQtYnk6IEFs ZXhhbmRlciBHb3JkZWV2IDxhZ29yZGVldkBsaW51eC5pYm0uY29tPgpTaWduZWQtb2ZmLWJ5OiBW YXNpbHkgR29yYmlrIDxnb3JAbGludXguaWJtLmNvbT4KLS0tCiBhcmNoL3MzOTAvaW5jbHVkZS9h c20vcGd0YWJsZS5oIHwgNDIgKysrKysrKysrKysrKysrKysrKysrKystLS0tLS0tLS0tCiBpbmNs dWRlL2xpbnV4L3BndGFibGUuaCAgICAgICAgIHwgMTAgKysrKysrKysKIG1tL2d1cC5jICAgICAg ICAgICAgICAgICAgICAgICAgfCAxOCArKysrKysrLS0tLS0tLQogMyBmaWxlcyBjaGFuZ2VkLCA0 OSBpbnNlcnRpb25zKCspLCAyMSBkZWxldGlvbnMoLSkKCmRpZmYgLS1naXQgYS9hcmNoL3MzOTAv aW5jbHVkZS9hc20vcGd0YWJsZS5oIGIvYXJjaC9zMzkwL2luY2x1ZGUvYXNtL3BndGFibGUuaApp bmRleCA3ZWIwMWE1NDU5Y2QuLmI1NTU2MWNjODc4NiAxMDA2NDQKLS0tIGEvYXJjaC9zMzkwL2lu Y2x1ZGUvYXNtL3BndGFibGUuaAorKysgYi9hcmNoL3MzOTAvaW5jbHVkZS9hc20vcGd0YWJsZS5o CkBAIC0xMjYwLDI2ICsxMjYwLDQ0IEBAIHN0YXRpYyBpbmxpbmUgcGdkX3QgKnBnZF9vZmZzZXRf cmF3KHBnZF90ICpwZ2QsIHVuc2lnbmVkIGxvbmcgYWRkcmVzcykKIAogI2RlZmluZSBwZ2Rfb2Zm c2V0KG1tLCBhZGRyZXNzKSBwZ2Rfb2Zmc2V0X3JhdyhSRUFEX09OQ0UoKG1tKS0+cGdkKSwgYWRk cmVzcykKIAotc3RhdGljIGlubGluZSBwNGRfdCAqcDRkX29mZnNldChwZ2RfdCAqcGdkLCB1bnNp Z25lZCBsb25nIGFkZHJlc3MpCitzdGF0aWMgaW5saW5lIHA0ZF90ICpwNGRfb2Zmc2V0X2xvY2ts ZXNzKHBnZF90ICpwZ2RwLCBwZ2RfdCBwZ2QsIHVuc2lnbmVkIGxvbmcgYWRkcmVzcykKIHsKLQlp ZiAoKHBnZF92YWwoKnBnZCkgJiBfUkVHSU9OX0VOVFJZX1RZUEVfTUFTSykgPj0gX1JFR0lPTl9F TlRSWV9UWVBFX1IxKQotCQlyZXR1cm4gKHA0ZF90ICopIHBnZF9kZXJlZigqcGdkKSArIHA0ZF9p bmRleChhZGRyZXNzKTsKLQlyZXR1cm4gKHA0ZF90ICopIHBnZDsKKwlpZiAoKHBnZF92YWwocGdk KSAmIF9SRUdJT05fRU5UUllfVFlQRV9NQVNLKSA+PSBfUkVHSU9OX0VOVFJZX1RZUEVfUjEpCisJ CXJldHVybiAocDRkX3QgKikgcGdkX2RlcmVmKHBnZCkgKyBwNGRfaW5kZXgoYWRkcmVzcyk7CisJ cmV0dXJuIChwNGRfdCAqKSBwZ2RwOwogfQorI2RlZmluZSBwNGRfb2Zmc2V0X2xvY2tsZXNzIHA0 ZF9vZmZzZXRfbG9ja2xlc3MKIAotc3RhdGljIGlubGluZSBwdWRfdCAqcHVkX29mZnNldChwNGRf dCAqcDRkLCB1bnNpZ25lZCBsb25nIGFkZHJlc3MpCitzdGF0aWMgaW5saW5lIHA0ZF90ICpwNGRf b2Zmc2V0KHBnZF90ICpwZ2RwLCB1bnNpZ25lZCBsb25nIGFkZHJlc3MpCiB7Ci0JaWYgKChwNGRf dmFsKCpwNGQpICYgX1JFR0lPTl9FTlRSWV9UWVBFX01BU0spID49IF9SRUdJT05fRU5UUllfVFlQ RV9SMikKLQkJcmV0dXJuIChwdWRfdCAqKSBwNGRfZGVyZWYoKnA0ZCkgKyBwdWRfaW5kZXgoYWRk cmVzcyk7Ci0JcmV0dXJuIChwdWRfdCAqKSBwNGQ7CisJcmV0dXJuIHA0ZF9vZmZzZXRfbG9ja2xl c3MocGdkcCwgKnBnZHAsIGFkZHJlc3MpOworfQorCitzdGF0aWMgaW5saW5lIHB1ZF90ICpwdWRf b2Zmc2V0X2xvY2tsZXNzKHA0ZF90ICpwNGRwLCBwNGRfdCBwNGQsIHVuc2lnbmVkIGxvbmcgYWRk cmVzcykKK3sKKwlpZiAoKHA0ZF92YWwocDRkKSAmIF9SRUdJT05fRU5UUllfVFlQRV9NQVNLKSA+ PSBfUkVHSU9OX0VOVFJZX1RZUEVfUjIpCisJCXJldHVybiAocHVkX3QgKikgcDRkX2RlcmVmKHA0 ZCkgKyBwdWRfaW5kZXgoYWRkcmVzcyk7CisJcmV0dXJuIChwdWRfdCAqKSBwNGRwOworfQorI2Rl ZmluZSBwdWRfb2Zmc2V0X2xvY2tsZXNzIHB1ZF9vZmZzZXRfbG9ja2xlc3MKKworc3RhdGljIGlu bGluZSBwdWRfdCAqcHVkX29mZnNldChwNGRfdCAqcDRkcCwgdW5zaWduZWQgbG9uZyBhZGRyZXNz KQoreworCXJldHVybiBwdWRfb2Zmc2V0X2xvY2tsZXNzKHA0ZHAsICpwNGRwLCBhZGRyZXNzKTsK IH0KICNkZWZpbmUgcHVkX29mZnNldCBwdWRfb2Zmc2V0CiAKLXN0YXRpYyBpbmxpbmUgcG1kX3Qg KnBtZF9vZmZzZXQocHVkX3QgKnB1ZCwgdW5zaWduZWQgbG9uZyBhZGRyZXNzKQorc3RhdGljIGlu bGluZSBwbWRfdCAqcG1kX29mZnNldF9sb2NrbGVzcyhwdWRfdCAqcHVkcCwgcHVkX3QgcHVkLCB1 bnNpZ25lZCBsb25nIGFkZHJlc3MpCit7CisJaWYgKChwdWRfdmFsKHB1ZCkgJiBfUkVHSU9OX0VO VFJZX1RZUEVfTUFTSykgPj0gX1JFR0lPTl9FTlRSWV9UWVBFX1IzKQorCQlyZXR1cm4gKHBtZF90 ICopIHB1ZF9kZXJlZihwdWQpICsgcG1kX2luZGV4KGFkZHJlc3MpOworCXJldHVybiAocG1kX3Qg KikgcHVkcDsKK30KKyNkZWZpbmUgcG1kX29mZnNldF9sb2NrbGVzcyBwbWRfb2Zmc2V0X2xvY2ts ZXNzCisKK3N0YXRpYyBpbmxpbmUgcG1kX3QgKnBtZF9vZmZzZXQocHVkX3QgKnB1ZHAsIHVuc2ln bmVkIGxvbmcgYWRkcmVzcykKIHsKLQlpZiAoKHB1ZF92YWwoKnB1ZCkgJiBfUkVHSU9OX0VOVFJZ X1RZUEVfTUFTSykgPj0gX1JFR0lPTl9FTlRSWV9UWVBFX1IzKQotCQlyZXR1cm4gKHBtZF90ICop IHB1ZF9kZXJlZigqcHVkKSArIHBtZF9pbmRleChhZGRyZXNzKTsKLQlyZXR1cm4gKHBtZF90ICop IHB1ZDsKKwlyZXR1cm4gcG1kX29mZnNldF9sb2NrbGVzcyhwdWRwLCAqcHVkcCwgYWRkcmVzcyk7 CiB9CiAjZGVmaW5lIHBtZF9vZmZzZXQgcG1kX29mZnNldAogCmRpZmYgLS1naXQgYS9pbmNsdWRl L2xpbnV4L3BndGFibGUuaCBiL2luY2x1ZGUvbGludXgvcGd0YWJsZS5oCmluZGV4IGU4Y2JjMmU3 OTVkNS4uZTg5OWQzNTA2NjcxIDEwMDY0NAotLS0gYS9pbmNsdWRlL2xpbnV4L3BndGFibGUuaAor KysgYi9pbmNsdWRlL2xpbnV4L3BndGFibGUuaApAQCAtMTQyNyw2ICsxNDI3LDE2IEBAIHR5cGVk ZWYgdW5zaWduZWQgaW50IHBndGJsX21vZF9tYXNrOwogI2RlZmluZSBtbV9wbWRfZm9sZGVkKG1t KQlfX2lzX2RlZmluZWQoX19QQUdFVEFCTEVfUE1EX0ZPTERFRCkKICNlbmRpZgogCisjaWZuZGVm IHA0ZF9vZmZzZXRfbG9ja2xlc3MKKyNkZWZpbmUgcDRkX29mZnNldF9sb2NrbGVzcyhwZ2RwLCBw Z2QsIGFkZHJlc3MpIHA0ZF9vZmZzZXQoJnBnZCwgYWRkcmVzcykKKyNlbmRpZgorI2lmbmRlZiBw dWRfb2Zmc2V0X2xvY2tsZXNzCisjZGVmaW5lIHB1ZF9vZmZzZXRfbG9ja2xlc3MocDRkcCwgcDRk LCBhZGRyZXNzKSBwdWRfb2Zmc2V0KCZwNGQsIGFkZHJlc3MpCisjZW5kaWYKKyNpZm5kZWYgcG1k X29mZnNldF9sb2NrbGVzcworI2RlZmluZSBwbWRfb2Zmc2V0X2xvY2tsZXNzKHB1ZHAsIHB1ZCwg YWRkcmVzcykgcG1kX29mZnNldCgmcHVkLCBhZGRyZXNzKQorI2VuZGlmCisKIC8qCiAgKiBwP2Rf bGVhZigpIC0gdHJ1ZSBpZiB0aGlzIGVudHJ5IGlzIGEgZmluYWwgbWFwcGluZyB0byBhIHBoeXNp Y2FsIGFkZHJlc3MuCiAgKiBUaGlzIGRpZmZlcnMgZnJvbSBwP2RfaHVnZSgpIGJ5IHRoZSBmYWN0 IHRoYXQgdGhleSBhcmUgYWx3YXlzIGF2YWlsYWJsZSAoaWYKZGlmZiAtLWdpdCBhL21tL2d1cC5j IGIvbW0vZ3VwLmMKaW5kZXggZTU3MzlhMTk3NGQ1Li41NzhiZjViZDhiZjggMTAwNjQ0Ci0tLSBh L21tL2d1cC5jCisrKyBiL21tL2d1cC5jCkBAIC0yNDg1LDEzICsyNDg1LDEzIEBAIHN0YXRpYyBp bnQgZ3VwX2h1Z2VfcGdkKHBnZF90IG9yaWcsIHBnZF90ICpwZ2RwLCB1bnNpZ25lZCBsb25nIGFk ZHIsCiAJcmV0dXJuIDE7CiB9CiAKLXN0YXRpYyBpbnQgZ3VwX3BtZF9yYW5nZShwdWRfdCBwdWQs IHVuc2lnbmVkIGxvbmcgYWRkciwgdW5zaWduZWQgbG9uZyBlbmQsCitzdGF0aWMgaW50IGd1cF9w bWRfcmFuZ2UocHVkX3QgKnB1ZHAsIHB1ZF90IHB1ZCwgdW5zaWduZWQgbG9uZyBhZGRyLCB1bnNp Z25lZCBsb25nIGVuZCwKIAkJdW5zaWduZWQgaW50IGZsYWdzLCBzdHJ1Y3QgcGFnZSAqKnBhZ2Vz LCBpbnQgKm5yKQogewogCXVuc2lnbmVkIGxvbmcgbmV4dDsKIAlwbWRfdCAqcG1kcDsKIAotCXBt ZHAgPSBwbWRfb2Zmc2V0KCZwdWQsIGFkZHIpOworCXBtZHAgPSBwbWRfb2Zmc2V0X2xvY2tsZXNz KHB1ZHAsIHB1ZCwgYWRkcik7CiAJZG8gewogCQlwbWRfdCBwbWQgPSBSRUFEX09OQ0UoKnBtZHAp OwogCkBAIC0yNTI4LDEzICsyNTI4LDEzIEBAIHN0YXRpYyBpbnQgZ3VwX3BtZF9yYW5nZShwdWRf dCBwdWQsIHVuc2lnbmVkIGxvbmcgYWRkciwgdW5zaWduZWQgbG9uZyBlbmQsCiAJcmV0dXJuIDE7 CiB9CiAKLXN0YXRpYyBpbnQgZ3VwX3B1ZF9yYW5nZShwNGRfdCBwNGQsIHVuc2lnbmVkIGxvbmcg YWRkciwgdW5zaWduZWQgbG9uZyBlbmQsCitzdGF0aWMgaW50IGd1cF9wdWRfcmFuZ2UocDRkX3Qg KnA0ZHAsIHA0ZF90IHA0ZCwgdW5zaWduZWQgbG9uZyBhZGRyLCB1bnNpZ25lZCBsb25nIGVuZCwK IAkJCSB1bnNpZ25lZCBpbnQgZmxhZ3MsIHN0cnVjdCBwYWdlICoqcGFnZXMsIGludCAqbnIpCiB7 CiAJdW5zaWduZWQgbG9uZyBuZXh0OwogCXB1ZF90ICpwdWRwOwogCi0JcHVkcCA9IHB1ZF9vZmZz ZXQoJnA0ZCwgYWRkcik7CisJcHVkcCA9IHB1ZF9vZmZzZXRfbG9ja2xlc3MocDRkcCwgcDRkLCBh ZGRyKTsKIAlkbyB7CiAJCXB1ZF90IHB1ZCA9IFJFQURfT05DRSgqcHVkcCk7CiAKQEAgLTI1NDks MjAgKzI1NDksMjAgQEAgc3RhdGljIGludCBndXBfcHVkX3JhbmdlKHA0ZF90IHA0ZCwgdW5zaWdu ZWQgbG9uZyBhZGRyLCB1bnNpZ25lZCBsb25nIGVuZCwKIAkJCWlmICghZ3VwX2h1Z2VfcGQoX19o dWdlcGQocHVkX3ZhbChwdWQpKSwgYWRkciwKIAkJCQkJIFBVRF9TSElGVCwgbmV4dCwgZmxhZ3Ms IHBhZ2VzLCBucikpCiAJCQkJcmV0dXJuIDA7Ci0JCX0gZWxzZSBpZiAoIWd1cF9wbWRfcmFuZ2Uo cHVkLCBhZGRyLCBuZXh0LCBmbGFncywgcGFnZXMsIG5yKSkKKwkJfSBlbHNlIGlmICghZ3VwX3Bt ZF9yYW5nZShwdWRwLCBwdWQsIGFkZHIsIG5leHQsIGZsYWdzLCBwYWdlcywgbnIpKQogCQkJcmV0 dXJuIDA7CiAJfSB3aGlsZSAocHVkcCsrLCBhZGRyID0gbmV4dCwgYWRkciAhPSBlbmQpOwogCiAJ cmV0dXJuIDE7CiB9CiAKLXN0YXRpYyBpbnQgZ3VwX3A0ZF9yYW5nZShwZ2RfdCBwZ2QsIHVuc2ln bmVkIGxvbmcgYWRkciwgdW5zaWduZWQgbG9uZyBlbmQsCitzdGF0aWMgaW50IGd1cF9wNGRfcmFu Z2UocGdkX3QgKnBnZHAsIHBnZF90IHBnZCwgdW5zaWduZWQgbG9uZyBhZGRyLCB1bnNpZ25lZCBs b25nIGVuZCwKIAkJCSB1bnNpZ25lZCBpbnQgZmxhZ3MsIHN0cnVjdCBwYWdlICoqcGFnZXMsIGlu dCAqbnIpCiB7CiAJdW5zaWduZWQgbG9uZyBuZXh0OwogCXA0ZF90ICpwNGRwOwogCi0JcDRkcCA9 IHA0ZF9vZmZzZXQoJnBnZCwgYWRkcik7CisJcDRkcCA9IHA0ZF9vZmZzZXRfbG9ja2xlc3MocGdk cCwgcGdkLCBhZGRyKTsKIAlkbyB7CiAJCXA0ZF90IHA0ZCA9IFJFQURfT05DRSgqcDRkcCk7CiAK QEAgLTI1NzQsNyArMjU3NCw3IEBAIHN0YXRpYyBpbnQgZ3VwX3A0ZF9yYW5nZShwZ2RfdCBwZ2Qs IHVuc2lnbmVkIGxvbmcgYWRkciwgdW5zaWduZWQgbG9uZyBlbmQsCiAJCQlpZiAoIWd1cF9odWdl X3BkKF9faHVnZXBkKHA0ZF92YWwocDRkKSksIGFkZHIsCiAJCQkJCSBQNERfU0hJRlQsIG5leHQs IGZsYWdzLCBwYWdlcywgbnIpKQogCQkJCXJldHVybiAwOwotCQl9IGVsc2UgaWYgKCFndXBfcHVk X3JhbmdlKHA0ZCwgYWRkciwgbmV4dCwgZmxhZ3MsIHBhZ2VzLCBucikpCisJCX0gZWxzZSBpZiAo IWd1cF9wdWRfcmFuZ2UocDRkcCwgcDRkLCBhZGRyLCBuZXh0LCBmbGFncywgcGFnZXMsIG5yKSkK IAkJCXJldHVybiAwOwogCX0gd2hpbGUgKHA0ZHArKywgYWRkciA9IG5leHQsIGFkZHIgIT0gZW5k KTsKIApAQCAtMjYwMiw3ICsyNjAyLDcgQEAgc3RhdGljIHZvaWQgZ3VwX3BnZF9yYW5nZSh1bnNp Z25lZCBsb25nIGFkZHIsIHVuc2lnbmVkIGxvbmcgZW5kLAogCQkJaWYgKCFndXBfaHVnZV9wZChf X2h1Z2VwZChwZ2RfdmFsKHBnZCkpLCBhZGRyLAogCQkJCQkgUEdESVJfU0hJRlQsIG5leHQsIGZs YWdzLCBwYWdlcywgbnIpKQogCQkJCXJldHVybjsKLQkJfSBlbHNlIGlmICghZ3VwX3A0ZF9yYW5n ZShwZ2QsIGFkZHIsIG5leHQsIGZsYWdzLCBwYWdlcywgbnIpKQorCQl9IGVsc2UgaWYgKCFndXBf cDRkX3JhbmdlKHBnZHAsIHBnZCwgYWRkciwgbmV4dCwgZmxhZ3MsIHBhZ2VzLCBucikpCiAJCQly ZXR1cm47CiAJfSB3aGlsZSAocGdkcCsrLCBhZGRyID0gbmV4dCwgYWRkciAhPSBlbmQpOwogfQot LSAK4qO/4qO/4qO/4qO/4qKL4qGA4qOA4qC54qO/4qO/4qO/4qO/CuKjv+Kjv+Kjv+Kjv+KgoOKj tuKhpuKggOKjv+Kjv+Kjv+Kjvwrio7/io7/io7/ioI/io7Tio67io7Tio6fioIjior/io7/io78K 4qO/4qO/4qGP4qKw4qO/4qCW4qOg4qO/4qGG4qCI4qO/4qO/CuKjv+Kim+KjteKjhOKgmeKjtuKj tuKhn+KjheKjoOKgueKjvwrio7/io5zio5vioLvioo7io4nio4nio4DioL/io6vio7Xio78KCl9f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCmxpbnV4LWFybS1r ZXJuZWwgbWFpbGluZyBsaXN0CmxpbnV4LWFybS1rZXJuZWxAbGlzdHMuaW5mcmFkZWFkLm9yZwpo dHRwOi8vbGlzdHMuaW5mcmFkZWFkLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2xpbnV4LWFybS1rZXJu ZWwK