From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on archive.lwn.net X-Spam-Level: X-Spam-Status: No, score=-5.6 required=5.0 tests=DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, T_DKIM_INVALID autolearn=ham autolearn_force=no version=3.4.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by archive.lwn.net (Postfix) with ESMTP id 303EE7D048 for ; Tue, 19 Jun 2018 08:26:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756545AbeFSI0I (ORCPT ); Tue, 19 Jun 2018 04:26:08 -0400 Received: from mail-co1nam03on0057.outbound.protection.outlook.com ([104.47.40.57]:50284 "EHLO NAM03-CO1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1756600AbeFSI0A (ORCPT ); Tue, 19 Jun 2018 04:26:00 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vmware.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fy6boBGdwuS5FZrb4JniTgAnOCVhYoeT0dVE2Jq2C9Y=; b=ZzGvh8MExoV5lKJ/eiwuhGTuLNeczdOyqhjIsG3RiHIzlIKvJbwKcqrDeQ3yzsDKSL1CMecKyf5t6tBHaJoyeVkExjHennyB8uAZTwpPhZB65plHvMFFCMUJ6ZZFzgwGuMZ2Yqs3MCGYiBs4nTTeH4EC6wzgUshqmBdgNojKtBI= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=thellstrom@vmware.com; Received: from localhost.vmware.com (155.4.205.56) by DM6PR05MB4585.namprd05.prod.outlook.com (2603:10b6:5:9f::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.884.16; Tue, 19 Jun 2018 08:25:47 +0000 From: Thomas Hellstrom To: dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org, peterz@infradead.org Cc: linux-graphics-maintainer@vmware.com, pv-drivers@vmware.com, Ingo Molnar , Jonathan Corbet , Gustavo Padovan , Maarten Lankhorst , Sean Paul , David Airlie , Davidlohr Bueso , "Paul E. McKenney" , Josh Triplett , Thomas Gleixner , Kate Stewart , Philippe Ombredanne , Greg Kroah-Hartman , linux-doc@vger.kernel.org, linux-media@vger.kernel.org, linaro-mm-sig@lists.linaro.org, Thomas Hellstrom Subject: [PATCH 1/3] locking: WW mutex cleanup Date: Tue, 19 Jun 2018 10:24:43 +0200 Message-Id: <20180619082445.11062-2-thellstrom@vmware.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180619082445.11062-1-thellstrom@vmware.com> References: <20180619082445.11062-1-thellstrom@vmware.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [155.4.205.56] X-ClientProxiedBy: CO2PR04CA0136.namprd04.prod.outlook.com (2603:10b6:104::14) To DM6PR05MB4585.namprd05.prod.outlook.com (2603:10b6:5:9f::22) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e3ece758-5c9c-4642-1d08-08d5d5be4571 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(7020095)(4652020)(4534165)(4627221)(201703031133081)(201702281549075)(5600026)(711020)(2017052603328)(7153060)(7193020);SRVR:DM6PR05MB4585; X-Microsoft-Exchange-Diagnostics: 1;DM6PR05MB4585;3:0quWA1GtpvLZGuKMvJa0ByRQjRVT+3IG5+vWh0czM3kYXCSZvK4+oxd5mcr3jmq937xapOs3qU1QD0vrmw+JBFddrwOH9iwB6o7ml2tl8/IBjNU2qj9YNmgOk6BGdUfyd0lFPTUYnfGGsPyIsuZsf0Ora3dhTbNNFL1+4SO2XCRWHljmmTNqylVSxHaMVnv+mvvyRd6sw0E/Z8g3OBFuvdPA5gNGe0WS7pocttdbYKvjHdzYMqTC7vuh6mFpArsT;25:ZYn/QBbIvXQGqO22ylHhDW7ieswljsS9+VgFtyZXOCrTHXOzJs7Ihb+wwgrPhfT87VflbEikd9uT7xl/t5Zc4ce6oyDqPb7xs9Alza2HysqT82H7mCubvx+siF9P4gjZd+cLu8MnN3zlklFD9CpXWSyjgd9vevlV1QbNIUxH90lH8XWWr5TIsqQ3vc+DMtDLTbBuxe55gtTIGSTssPpMuETAnpkoeNm93kqvGjUP6js4xliigynF5JdSqOT7MzPQThEWtVhFgN2C6+C2reQv9OYqu8ttpiX1OqBIfBnCq3EcvgzpmQCoqAqjrDcrS/tz6vEIWqifvz8LdkjQTp7cSQ==;31:rR9m8JQXXcbnE+3+Ay8rhTLM30PgJY3tpBr37IBXyjswbG0osKmLucQmjbTnkKJ6/lr+BAPeg9H9rAjSN4+YnutC5vr76nQvNeo6rTaK+D0a0mvRsjlEjWv/o26IKkBNbHwpgiUrTrw6ei4EReZ6MSs8oZn+tGjQehb8NcnQPAequf/Gr3w0yWObv8o9L7ZW51lDMWKTfnbTGDQm2Mqz3I3jarZTmY8N3bcPy/J/xVY= X-MS-TrafficTypeDiagnostic: DM6PR05MB4585: X-LD-Processed: b39138ca-3cee-4b4a-a4d6-cd83d9dd62f0,ExtAddr X-Microsoft-Exchange-Diagnostics: 1;DM6PR05MB4585;20:CcEXPbTcmlYDb6a50mjBtNbXpULGz+SUACZmuC1B3XzS9JP1TllW7Pa6OwlXskWRrcSxiyIWGVLtryEio2Tq4Xpz35sFr4q1ipWBSrJqicdqAVdn0DFb1atcVARswm0q/305jiNIt7az1q3VANYWuNtSfsih+J72uxE7x4jFil3xdWclKmxTHnCf8vyddOCyvrhwJbtByRiQ2XLniu+ZyV2l3VMN1BV/a/X1xZegus+ujxWnathPZTd/cSzu6hD1gMPIbl7bFuovswcIw77Y/Ap1/Zk4x/2mzYvhUs2Ax4ol9sq9Mtb+pf0Dhowk2lKzZdgd2ob6dmdQ8J9S7wivF65hoydyXSI61zqfoc2q4hhZwXhdlbjJ9EPJhcbudEgRnEt4s4/pWRsNvSF8NnBr0gOCMkl8NsGx5/aA9zgr85X1N57dMiOwXV5dbUt7qH+XjuWPPCe+DXgEWQAJjVypyMhhxFlUsKCGfgc0qOManr8+ibR5anR2r/yWPJxjKUjV X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(61668805478150)(9452136761055)(1553240931313)(104084551191319)(228905959029699); X-MS-Exchange-SenderADCheck: 1 X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(93006095)(93001095)(10201501046)(3231254)(944501410)(52105095)(3002001)(149027)(150027)(6041310)(20161123564045)(20161123558120)(20161123560045)(20161123562045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(6072148)(201708071742011)(7699016);SRVR:DM6PR05MB4585;BCL:0;PCL:0;RULEID:;SRVR:DM6PR05MB4585; X-Microsoft-Exchange-Diagnostics: 1;DM6PR05MB4585;4:GNdkLAGLX6fv1xFgpNMW5gnvFQvnYles70lksDFjUReFixubt4XM1KJFUKiFpI9eqrccxbZ4fHign+MkWOAwuzJNzFee0Mi7vVV2QyrsaEXUE2Ec77pabMoOEaBjrxO8GW40NCU2/+1Kgx0bYBFyDFDETrh7vNFDOjFIhXxfQKqPNzjUHm2q+lADPi7yRfJaneRcds5FBmHUfdJEaS35ax49xnNIqfAhmH4dwGYxdV5rqbdg6J+iblwOJo2xa4Vea8ptLmxYu0eq44YuwlYlwiw6/t6zDGLtA6WRvKvMC4pwBl/I4p/GcKslo8Wlyv2CJEbLL+9qOaDVeu+/v5lvksW+JzV7qNISfAp9jc2WtxPHGQwz6GtbrVMwhzZeTa8lnMpFfz7EwX287hioj4vZYOVS0QSDzKaKCwtWiVD59BszRqGGCIuxYEvINPkJW4KP X-Forefront-PRVS: 07083FF734 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(6069001)(39380400002)(346002)(396003)(39860400002)(376002)(366004)(199004)(189003)(106356001)(68736007)(76506005)(53416004)(956004)(105586002)(11346002)(7736002)(305945005)(107886003)(6116002)(53936002)(316002)(54906003)(3846002)(69596002)(478600001)(486006)(50226002)(66066001)(2906002)(186003)(16526019)(48376002)(47776003)(5660300001)(26005)(97736004)(6486002)(8676002)(7696005)(52116002)(86362001)(76176011)(51416003)(8936002)(476003)(2616005)(16586007)(25786009)(1076002)(59450400001)(446003)(4326008)(6666003)(81156014)(81166006)(7416002)(551934003)(36756003)(50466002)(386003);DIR:OUT;SFP:1101;SCL:1;SRVR:DM6PR05MB4585;H:localhost.vmware.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;A:1;MX:1; Received-SPF: None (protection.outlook.com: vmware.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;DM6PR05MB4585;23:ZN+jjEIFSBt9Vw/8isAj/B3RoMP2pwwikTcybeBEw?= =?us-ascii?Q?aoXsAu8teriYq7+bP3hqwUb16/6AWw5j+88vNcxgmvYQ0myhhS6Ex2Wx68uy?= =?us-ascii?Q?qw+/3jDbUOTnx5wLiscyutLNJEuEbMig9Ft+5kdCRM5WfLjJsauWV7ZIYUvW?= =?us-ascii?Q?sltKjVq8ty3yFLfWr2mhPYccGnh9tgUC7O/zG6uUo3+LRIrp/yWW9cBBu2J7?= =?us-ascii?Q?HmhCS0mxWKkfFeiqHksdiFXQ/M2j8zvp15NCRc8uPfEKtS4m3+qCPt4BIRDU?= =?us-ascii?Q?/u4hVJlNQNDpLkclreH2m3GP8A8VpreI+Hm09GkSaUG71iQEJsqN8BZyTzl9?= =?us-ascii?Q?Nl4BSn0lsV8lEDaMXRXkjYJywLSszls4DZvNIyOkEg5M2Bjxn8z6r38vM6DF?= =?us-ascii?Q?5BE22GO4IurtfVoGJ7OA6r1JrlzTYSVGKiuZxvhXrKAJLLuzsJLmI9PVUXzF?= =?us-ascii?Q?LZGSWnp0u3IXRAcgZG718IWu7FeiY49JvbC6pNCcBDmDIy5a/j6QgBrgr5Mq?= =?us-ascii?Q?cIExfyWxmeR75AmSFM26IqhSVdK4AifYndcFF5ZLWdda5GOjISNHlFVnfzkZ?= =?us-ascii?Q?/ePnO1H0Zs/1gJCxqjp0Nu2tzz7Pajq/uIEJv6aewe1m4cHd3huRZxAc9e1q?= =?us-ascii?Q?0MfYxNjLRklw/QKJXiGLa80oWiqKPS23f4Uv7G5RKrFvPd5/HecIbMTfM4vL?= =?us-ascii?Q?RE5THw4dV9YKyTXpjP5vd5x1st/Dyg/g4jcc4joPXGjhW6JlKUAKwvN9xYul?= =?us-ascii?Q?nIogBUIt63WMCChHS2GHXASMzB7J8E1MfhL4ObJD/0l0KjcJhSPZm7i9n/Uz?= =?us-ascii?Q?3ok45FF1R9TbPwTyQeVTRYN9CUMBCk8lldF4TJ4fKh+iRP0pU36Y7n4A6w1N?= =?us-ascii?Q?qYYidSfJV38Bbe2qCTtZz0k16wtWfAUejkHk9Eplwp9+2LmED5fCoJkB4Ahx?= =?us-ascii?Q?8p5N7r8CMIa35p39D0N7akbvrk8/ufiJWIWdw0p06kovYuZKGDjTiPL7qLDg?= =?us-ascii?Q?DOCCRpUitUrY0vilZAtaaPU24XVI615Zz6g7IMHt+bdAwA3yiHVdGMCWgWQe?= =?us-ascii?Q?KFhD0ImwckPdI7+wmLhgHT5xwz83tUKW1ehseuftoqAKrRPZea0JnEKueVJe?= =?us-ascii?Q?4qYGb1zczJ9vcyzyZTKVtZ8my4v04vVZLPscMByBsvvyabEBQcNeZTG3uIsy?= =?us-ascii?Q?Dibnjo4PKcJaD2G/GIP5g0W6Xs6qZxO0aNP8TMJOCQm6KzKvQ0Lh8zDhDNSe?= =?us-ascii?Q?jJWlELfk4wUGj1DFzbIFVd/nDwMj9CGg4Q5E0FXcBoYC0HRsp0FXM8B4rrqI?= =?us-ascii?Q?NNLYtLgvd05JWTY4l8ifXIyeVuRaFnxhU49bfcfJgW5KwyqtVTSZHcGfftF7?= =?us-ascii?Q?Q5QBA=3D=3D?= X-Microsoft-Antispam-Message-Info: GNfSJONmuQMOiN7Aq1ECCNtXdbYYl94dze06GqrtTAyHt+hZh7UW/0/kGkj2ZF8D3bNBLfzBIUnrWBZDGTdxs68LmuOuONpKKuEQQZutPFF3Ily6bgeInT+tkXysLsquIZQ+t2oblpTfns7cqryCdQe+ZeVfIgL7ocIbk0Ytdwx3biqrqjH0O+RSgild6usKOx9ujfmTdi8fwDKZuePGuDibwX/FYDYN+s+T4m/SSb5xvCwAelhK0lvKcKR9N5xK9YPTW0U1SwShtB6ywrIOdEJHru0IWoymeiF2j5KHuBD+SK66bOrmPPHQz7ichHdRI5KL/0NYsgOEUukYeLPW4w== X-Microsoft-Exchange-Diagnostics: 1;DM6PR05MB4585;6:GPZnTKwd+PoF79oFPhwyV/dKtp8fS1cb0rY0+WS0mcGUbEiUtVlrsYG7SQYc0m3VnUqh5NG5hXJQRcUk0UxGG542KP+4mN1qxT8EhBmUdLWmkc1KUwRr+nINfnSRfeKlbyd8m8cTt09XSUIMbNRXJvZ0CAEebzu2YV2ebyV5rpBKd4uHfubW3MxMY9PvXUmMgknINzv8LT/z5ZAXCB0oxrPJFhvkocnwIBxo7SbgfCJ5Eh9ZoINKq0zyqCobaAxCvKTVBmAUItLfFrtakayBkctp6P/GAbaduBwFm8c9piXRUMYnTMPPuWbAJYSrIN7dvF72B/B2ygSs1HBkYnJw/+RqjY7xIn4Q97lJtFsU2tKi4Owu/aCDKqT9vVaT2a9ASNY460X6cEkOFJpn/wdo6ftcDBRN1w0dh1CuT3qzlgh7W9w5oPOKUp/eUKrv8jF6x1Ha5s3j47Q8kXsOIKqWKg==;5:3/+xpSKd5jjyqrphqEJeKVGZ7p8LKUghxnIgm/5uOg8TWrSfD/wUUaByfm6unQHDkVZm0ByIWgdw3SNrnaYXsffL00q1jbjVdDxAnTRDHIiiYWJJFMKskoKbj+cKzt9QVO/oTcImokMaW4jEvcQGMRQox58BIQ8k+WQPCiPRbPE=;24:vZ3Js5EOQPzpNOqui1s9udQdEwIIH6Lm01rZa24Slo/oi4zeHKFF84A92vtT6gqi8BjveRKoow5ubNhhMlWeIHTG9Ze1ymLpj7PDodLtEo4= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;DM6PR05MB4585;7:RuzJxbdTk1CYl5I30j+CYmhEozx2QkLL4AoL8IV159JZi0Nsbrh78RwnLgapOKrhNIRqBmgbPyfyMMflHTsKqwjTPbfitEPokFiJOSKKhdQTgOf0hsjS9sCSc01T3gAdA6kDze+fBs3bSogmtSAVTfs4ONmyVPTvYaxeNsbieOIOn3knaebdap2tUUOIGUf7MalCgizezolvfOmaCSLO6eaahYn1ix9KcoepVAXws327knRdpBnE+n9alhNBFC6D;20:6bFzxmO31jvABFTThQbuLr2JTNg7bnD8RaySQFU5o41e48S9V9kgB1041V/YxigJCIEtPISp3WSen04zOjft6rBhK9Lb/LInOKKn55+rxSjJHKxwO8GwVGAk5PgKirMiZODqP5cfeqGbH8EmIIq6uaJwFEsCLmMeYfpZB+wpRpo= X-OriginatorOrg: vmware.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jun 2018 08:25:47.7650 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e3ece758-5c9c-4642-1d08-08d5d5be4571 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: b39138ca-3cee-4b4a-a4d6-cd83d9dd62f0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR05MB4585 Sender: linux-doc-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-doc@vger.kernel.org From: Peter Ziljstra Make the WW mutex code more readable by adding comments, splitting up functions and pointing out that we're actually using the Wait-Die algorithm. Cc: Ingo Molnar Cc: Jonathan Corbet Cc: Gustavo Padovan Cc: Maarten Lankhorst Cc: Sean Paul Cc: David Airlie Cc: Davidlohr Bueso Cc: "Paul E. McKenney" Cc: Josh Triplett Cc: Thomas Gleixner Cc: Kate Stewart Cc: Philippe Ombredanne Cc: Greg Kroah-Hartman Cc: linux-doc@vger.kernel.org Cc: linux-media@vger.kernel.org Cc: linaro-mm-sig@lists.linaro.org Co-authored-by: Thomas Hellstrom Signed-off-by: Thomas Hellstrom --- Documentation/locking/ww-mutex-design.txt | 12 +- include/linux/ww_mutex.h | 28 ++--- kernel/locking/mutex.c | 202 ++++++++++++++++++------------ 3 files changed, 145 insertions(+), 97 deletions(-) diff --git a/Documentation/locking/ww-mutex-design.txt b/Documentation/locking/ww-mutex-design.txt index 34c3a1b50b9a..2fd7f2a2af21 100644 --- a/Documentation/locking/ww-mutex-design.txt +++ b/Documentation/locking/ww-mutex-design.txt @@ -32,10 +32,10 @@ the oldest task) wins, and the one with the higher reservation id (i.e. the younger task) unlocks all of the buffers that it has already locked, and then tries again. -In the RDBMS literature this deadlock handling approach is called wait/wound: +In the RDBMS literature this deadlock handling approach is called wait/die: The older tasks waits until it can acquire the contended lock. The younger tasks needs to back off and drop all the locks it is currently holding, i.e. the -younger task is wounded. +younger task dies. Concepts -------- @@ -56,9 +56,9 @@ Furthermore there are three different class of w/w lock acquire functions: * Normal lock acquisition with a context, using ww_mutex_lock. -* Slowpath lock acquisition on the contending lock, used by the wounded task - after having dropped all already acquired locks. These functions have the - _slow postfix. +* Slowpath lock acquisition on the contending lock, used by the task that just + killed its transaction after having dropped all already acquired locks. + These functions have the _slow postfix. From a simple semantics point-of-view the _slow functions are not strictly required, since simply calling the normal ww_mutex_lock functions on the @@ -220,7 +220,7 @@ mutexes are a natural fit for such a case for two reasons: Note that this approach differs in two important ways from the above methods: - Since the list of objects is dynamically constructed (and might very well be - different when retrying due to hitting the -EDEADLK wound condition) there's + different when retrying due to hitting the -EDEADLK die condition) there's no need to keep any object on a persistent list when it's not locked. We can therefore move the list_head into the object itself. - On the other hand the dynamic object list construction also means that the -EALREADY return diff --git a/include/linux/ww_mutex.h b/include/linux/ww_mutex.h index 39fda195bf78..f82fce2229c8 100644 --- a/include/linux/ww_mutex.h +++ b/include/linux/ww_mutex.h @@ -6,7 +6,7 @@ * * Copyright (C) 2004, 2005, 2006 Red Hat, Inc., Ingo Molnar * - * Wound/wait implementation: + * Wait/Die implementation: * Copyright (C) 2013 Canonical Ltd. * * This file contains the main data structure and API definitions. @@ -28,9 +28,9 @@ struct ww_class { struct ww_acquire_ctx { struct task_struct *task; unsigned long stamp; - unsigned acquired; + unsigned int acquired; #ifdef CONFIG_DEBUG_MUTEXES - unsigned done_acquire; + unsigned int done_acquire; struct ww_class *ww_class; struct ww_mutex *contending_lock; #endif @@ -38,8 +38,8 @@ struct ww_acquire_ctx { struct lockdep_map dep_map; #endif #ifdef CONFIG_DEBUG_WW_MUTEX_SLOWPATH - unsigned deadlock_inject_interval; - unsigned deadlock_inject_countdown; + unsigned int deadlock_inject_interval; + unsigned int deadlock_inject_countdown; #endif }; @@ -102,7 +102,7 @@ static inline void ww_mutex_init(struct ww_mutex *lock, * * Context-based w/w mutex acquiring can be done in any order whatsoever within * a given lock class. Deadlocks will be detected and handled with the - * wait/wound logic. + * wait/die logic. * * Mixing of context-based w/w mutex acquiring and single w/w mutex locking can * result in undetected deadlocks and is so forbidden. Mixing different contexts @@ -195,13 +195,13 @@ static inline void ww_acquire_fini(struct ww_acquire_ctx *ctx) * Lock the w/w mutex exclusively for this task. * * Deadlocks within a given w/w class of locks are detected and handled with the - * wait/wound algorithm. If the lock isn't immediately avaiable this function + * wait/die algorithm. If the lock isn't immediately available this function * will either sleep until it is (wait case). Or it selects the current context - * for backing off by returning -EDEADLK (wound case). Trying to acquire the + * for backing off by returning -EDEADLK (die case). Trying to acquire the * same lock with the same context twice is also detected and signalled by * returning -EALREADY. Returns 0 if the mutex was successfully acquired. * - * In the wound case the caller must release all currently held w/w mutexes for + * In the die case the caller must release all currently held w/w mutexes for * the given context and then wait for this contending lock to be available by * calling ww_mutex_lock_slow. Alternatively callers can opt to not acquire this * lock and proceed with trying to acquire further w/w mutexes (e.g. when @@ -226,14 +226,14 @@ extern int /* __must_check */ ww_mutex_lock(struct ww_mutex *lock, struct ww_acq * Lock the w/w mutex exclusively for this task. * * Deadlocks within a given w/w class of locks are detected and handled with the - * wait/wound algorithm. If the lock isn't immediately avaiable this function + * wait/die algorithm. If the lock isn't immediately available this function * will either sleep until it is (wait case). Or it selects the current context - * for backing off by returning -EDEADLK (wound case). Trying to acquire the + * for backing off by returning -EDEADLK (die case). Trying to acquire the * same lock with the same context twice is also detected and signalled by * returning -EALREADY. Returns 0 if the mutex was successfully acquired. If a * signal arrives while waiting for the lock then this function returns -EINTR. * - * In the wound case the caller must release all currently held w/w mutexes for + * In the die case the caller must release all currently held w/w mutexes for * the given context and then wait for this contending lock to be available by * calling ww_mutex_lock_slow_interruptible. Alternatively callers can opt to * not acquire this lock and proceed with trying to acquire further w/w mutexes @@ -256,7 +256,7 @@ extern int __must_check ww_mutex_lock_interruptible(struct ww_mutex *lock, * @lock: the mutex to be acquired * @ctx: w/w acquire context * - * Acquires a w/w mutex with the given context after a wound case. This function + * Acquires a w/w mutex with the given context after a die case. This function * will sleep until the lock becomes available. * * The caller must have released all w/w mutexes already acquired with the @@ -290,7 +290,7 @@ ww_mutex_lock_slow(struct ww_mutex *lock, struct ww_acquire_ctx *ctx) * @lock: the mutex to be acquired * @ctx: w/w acquire context * - * Acquires a w/w mutex with the given context after a wound case. This function + * Acquires a w/w mutex with the given context after a die case. This function * will sleep until the lock becomes available and returns 0 when the lock has * been acquired. If a signal arrives while waiting for the lock then this * function returns -EINTR. diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c index 2048359f33d2..412b4fc08235 100644 --- a/kernel/locking/mutex.c +++ b/kernel/locking/mutex.c @@ -243,6 +243,17 @@ void __sched mutex_lock(struct mutex *lock) EXPORT_SYMBOL(mutex_lock); #endif +/* + * Wait-Die: + * The newer transactions are killed when: + * It (the new transaction) makes a request for a lock being held + * by an older transaction. + */ + +/* + * Associate the ww_mutex @ww with the context @ww_ctx under which we acquired + * it. + */ static __always_inline void ww_mutex_lock_acquired(struct ww_mutex *ww, struct ww_acquire_ctx *ww_ctx) { @@ -281,26 +292,53 @@ ww_mutex_lock_acquired(struct ww_mutex *ww, struct ww_acquire_ctx *ww_ctx) DEBUG_LOCKS_WARN_ON(ww_ctx->ww_class != ww->ww_class); #endif ww_ctx->acquired++; + ww->ctx = ww_ctx; } +/* + * Determine if context @a is 'after' context @b. IOW, @a is a younger + * transaction than @b and depending on algorithm either needs to wait for + * @b or die. + */ static inline bool __sched __ww_ctx_stamp_after(struct ww_acquire_ctx *a, struct ww_acquire_ctx *b) { - return a->stamp - b->stamp <= LONG_MAX && - (a->stamp != b->stamp || a > b); + + return (signed long)(a->stamp - b->stamp) > 0; +} + +/* + * Wait-Die; wake a younger waiter context (when locks held) such that it can + * die. + * + * Among waiters with context, only the first one can have other locks acquired + * already (ctx->acquired > 0), because __ww_mutex_add_waiter() and + * __ww_mutex_check_kill() wake any but the earliest context. + */ +static bool __sched +__ww_mutex_die(struct mutex *lock, struct mutex_waiter *waiter, + struct ww_acquire_ctx *ww_ctx) +{ + if (waiter->ww_ctx->acquired > 0 && + __ww_ctx_stamp_after(waiter->ww_ctx, ww_ctx)) { + debug_mutex_wake_waiter(lock, waiter); + wake_up_process(waiter->task); + } + + return true; } /* - * Wake up any waiters that may have to back off when the lock is held by the - * given context. + * We just acquired @lock under @ww_ctx, if there are later contexts waiting + * behind us on the wait-list, check if they need to die. * - * Due to the invariants on the wait list, this can only affect the first - * waiter with a context. + * See __ww_mutex_add_waiter() for the list-order construction; basically the + * list is ordered by stamp, smallest (oldest) first. * * The current task must not be on the wait list. */ static void __sched -__ww_mutex_wakeup_for_backoff(struct mutex *lock, struct ww_acquire_ctx *ww_ctx) +__ww_mutex_check_waiters(struct mutex *lock, struct ww_acquire_ctx *ww_ctx) { struct mutex_waiter *cur; @@ -310,30 +348,23 @@ __ww_mutex_wakeup_for_backoff(struct mutex *lock, struct ww_acquire_ctx *ww_ctx) if (!cur->ww_ctx) continue; - if (cur->ww_ctx->acquired > 0 && - __ww_ctx_stamp_after(cur->ww_ctx, ww_ctx)) { - debug_mutex_wake_waiter(lock, cur); - wake_up_process(cur->task); - } - - break; + if (__ww_mutex_die(lock, cur, ww_ctx)) + break; } } /* - * After acquiring lock with fastpath or when we lost out in contested - * slowpath, set ctx and wake up any waiters so they can recheck. + * After acquiring lock with fastpath, where we do not hold wait_lock, set ctx + * and wake up any waiters so they can recheck. */ static __always_inline void ww_mutex_set_context_fastpath(struct ww_mutex *lock, struct ww_acquire_ctx *ctx) { ww_mutex_lock_acquired(lock, ctx); - lock->ctx = ctx; - /* * The lock->ctx update should be visible on all cores before - * the atomic read is done, otherwise contended waiters might be + * the WAITERS check is done, otherwise contended waiters might be * missed. The contended waiters will either see ww_ctx == NULL * and keep spinning, or it will acquire wait_lock, add itself * to waiter list and sleep. @@ -347,29 +378,14 @@ ww_mutex_set_context_fastpath(struct ww_mutex *lock, struct ww_acquire_ctx *ctx) return; /* - * Uh oh, we raced in fastpath, wake up everyone in this case, - * so they can see the new lock->ctx. + * Uh oh, we raced in fastpath, check if any of the waiters need to + * die. */ spin_lock(&lock->base.wait_lock); - __ww_mutex_wakeup_for_backoff(&lock->base, ctx); + __ww_mutex_check_waiters(&lock->base, ctx); spin_unlock(&lock->base.wait_lock); } -/* - * After acquiring lock in the slowpath set ctx. - * - * Unlike for the fast path, the caller ensures that waiters are woken up where - * necessary. - * - * Callers must hold the mutex wait_lock. - */ -static __always_inline void -ww_mutex_set_context_slowpath(struct ww_mutex *lock, struct ww_acquire_ctx *ctx) -{ - ww_mutex_lock_acquired(lock, ctx); - lock->ctx = ctx; -} - #ifdef CONFIG_MUTEX_SPIN_ON_OWNER static inline @@ -645,37 +661,73 @@ void __sched ww_mutex_unlock(struct ww_mutex *lock) } EXPORT_SYMBOL(ww_mutex_unlock); + +static __always_inline int __sched +__ww_mutex_kill(struct mutex *lock, struct ww_acquire_ctx *ww_ctx) +{ + if (ww_ctx->acquired > 0) { +#ifdef CONFIG_DEBUG_MUTEXES + struct ww_mutex *ww; + + ww = container_of(lock, struct ww_mutex, base); + DEBUG_LOCKS_WARN_ON(ww_ctx->contending_lock); + ww_ctx->contending_lock = ww; +#endif + return -EDEADLK; + } + + return 0; +} + + +/* + * Check whether we need to kill the transaction for the current lock acquire. + * + * Wait-Die: If we're trying to acquire a lock already held by an older + * context, kill ourselves. + * + * Since __ww_mutex_add_waiter() orders the wait-list on stamp, we only have to + * look at waiters before us in the wait-list. + */ static inline int __sched -__ww_mutex_lock_check_stamp(struct mutex *lock, struct mutex_waiter *waiter, - struct ww_acquire_ctx *ctx) +__ww_mutex_check_kill(struct mutex *lock, struct mutex_waiter *waiter, + struct ww_acquire_ctx *ctx) { struct ww_mutex *ww = container_of(lock, struct ww_mutex, base); struct ww_acquire_ctx *hold_ctx = READ_ONCE(ww->ctx); struct mutex_waiter *cur; + if (ctx->acquired == 0) + return 0; + if (hold_ctx && __ww_ctx_stamp_after(ctx, hold_ctx)) - goto deadlock; + return __ww_mutex_kill(lock, ctx); /* * If there is a waiter in front of us that has a context, then its - * stamp is earlier than ours and we must back off. + * stamp is earlier than ours and we must kill ourself. */ cur = waiter; list_for_each_entry_continue_reverse(cur, &lock->wait_list, list) { - if (cur->ww_ctx) - goto deadlock; + if (!cur->ww_ctx) + continue; + + return __ww_mutex_kill(lock, ctx); } return 0; - -deadlock: -#ifdef CONFIG_DEBUG_MUTEXES - DEBUG_LOCKS_WARN_ON(ctx->contending_lock); - ctx->contending_lock = ww; -#endif - return -EDEADLK; } +/* + * Add @waiter to the wait-list, keep the wait-list ordered by stamp, smallest + * first. Such that older contexts are preferred to acquire the lock over + * younger contexts. + * + * Waiters without context are interspersed in FIFO order. + * + * Furthermore, for Wait-Die kill ourself immediately when possible (there are + * older contexts already waiting) to avoid unnecessary waiting. + */ static inline int __sched __ww_mutex_add_waiter(struct mutex_waiter *waiter, struct mutex *lock, @@ -692,7 +744,7 @@ __ww_mutex_add_waiter(struct mutex_waiter *waiter, /* * Add the waiter before the first waiter with a higher stamp. * Waiters without a context are skipped to avoid starving - * them. + * them. Wait-Die waiters may die here. */ pos = &lock->wait_list; list_for_each_entry_reverse(cur, &lock->wait_list, list) { @@ -700,34 +752,27 @@ __ww_mutex_add_waiter(struct mutex_waiter *waiter, continue; if (__ww_ctx_stamp_after(ww_ctx, cur->ww_ctx)) { - /* Back off immediately if necessary. */ - if (ww_ctx->acquired > 0) { -#ifdef CONFIG_DEBUG_MUTEXES - struct ww_mutex *ww; + /* + * Wait-Die: if we find an older context waiting, there + * is no point in queueing behind it, as we'd have to + * die the moment it would acquire the lock. + */ + int ret = __ww_mutex_kill(lock, ww_ctx); - ww = container_of(lock, struct ww_mutex, base); - DEBUG_LOCKS_WARN_ON(ww_ctx->contending_lock); - ww_ctx->contending_lock = ww; -#endif - return -EDEADLK; - } + if (ret) + return ret; break; } pos = &cur->list; - /* - * Wake up the waiter so that it gets a chance to back - * off. - */ - if (cur->ww_ctx->acquired > 0) { - debug_mutex_wake_waiter(lock, cur); - wake_up_process(cur->task); - } + /* Wait-Die: ensure younger waiters die. */ + __ww_mutex_die(lock, cur, ww_ctx); } list_add_tail(&waiter->list, pos); + return 0; } @@ -771,7 +816,7 @@ __mutex_lock_common(struct mutex *lock, long state, unsigned int subclass, */ if (__mutex_trylock(lock)) { if (use_ww_ctx && ww_ctx) - __ww_mutex_wakeup_for_backoff(lock, ww_ctx); + __ww_mutex_check_waiters(lock, ww_ctx); goto skip_wait; } @@ -789,10 +834,13 @@ __mutex_lock_common(struct mutex *lock, long state, unsigned int subclass, waiter.ww_ctx = MUTEX_POISON_WW_CTX; #endif } else { - /* Add in stamp order, waking up waiters that must back off. */ + /* + * Add in stamp order, waking up waiters that must kill + * themselves. + */ ret = __ww_mutex_add_waiter(&waiter, lock, ww_ctx); if (ret) - goto err_early_backoff; + goto err_early_kill; waiter.ww_ctx = ww_ctx; } @@ -814,7 +862,7 @@ __mutex_lock_common(struct mutex *lock, long state, unsigned int subclass, goto acquired; /* - * Check for signals and wound conditions while holding + * Check for signals and kill conditions while holding * wait_lock. This ensures the lock cancellation is ordered * against mutex_unlock() and wake-ups do not go missing. */ @@ -823,8 +871,8 @@ __mutex_lock_common(struct mutex *lock, long state, unsigned int subclass, goto err; } - if (use_ww_ctx && ww_ctx && ww_ctx->acquired > 0) { - ret = __ww_mutex_lock_check_stamp(lock, &waiter, ww_ctx); + if (use_ww_ctx && ww_ctx) { + ret = __ww_mutex_check_kill(lock, &waiter, ww_ctx); if (ret) goto err; } @@ -869,7 +917,7 @@ __mutex_lock_common(struct mutex *lock, long state, unsigned int subclass, lock_acquired(&lock->dep_map, ip); if (use_ww_ctx && ww_ctx) - ww_mutex_set_context_slowpath(ww, ww_ctx); + ww_mutex_lock_acquired(ww, ww_ctx); spin_unlock(&lock->wait_lock); preempt_enable(); @@ -878,7 +926,7 @@ __mutex_lock_common(struct mutex *lock, long state, unsigned int subclass, err: __set_current_state(TASK_RUNNING); mutex_remove_waiter(lock, &waiter, current); -err_early_backoff: +err_early_kill: spin_unlock(&lock->wait_lock); debug_mutex_free_waiter(&waiter); mutex_release(&lock->dep_map, 1, ip); -- 2.14.3 -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html From mboxrd@z Thu Jan 1 00:00:00 1970 From: Thomas Hellstrom Subject: [PATCH 1/3] locking: WW mutex cleanup Date: Tue, 19 Jun 2018 10:24:43 +0200 Message-ID: <20180619082445.11062-2-thellstrom@vmware.com> References: <20180619082445.11062-1-thellstrom@vmware.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Received: from NAM05-DM3-obe.outbound.protection.outlook.com (mail-dm3nam05on062d.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe51::62d]) by gabe.freedesktop.org (Postfix) with ESMTPS id 351E46E4EF for ; Tue, 19 Jun 2018 08:26:00 +0000 (UTC) In-Reply-To: <20180619082445.11062-1-thellstrom@vmware.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" To: dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org, peterz@infradead.org Cc: Kate Stewart , Thomas Hellstrom , Davidlohr Bueso , Jonathan Corbet , pv-drivers@vmware.com, linux-doc@vger.kernel.org, Josh Triplett , linaro-mm-sig@lists.linaro.org, David Airlie , Greg Kroah-Hartman , Ingo Molnar , linux-graphics-maintainer@vmware.com, Philippe Ombredanne , Thomas Gleixner , "Paul E. McKenney" , linux-media@vger.kernel.org List-Id: dri-devel@lists.freedesktop.org RnJvbTogUGV0ZXIgWmlsanN0cmEgPHBldGVyekBpbmZyYWRlYWQub3JnPgoKTWFrZSB0aGUgV1cg bXV0ZXggY29kZSBtb3JlIHJlYWRhYmxlIGJ5IGFkZGluZyBjb21tZW50cywgc3BsaXR0aW5nIHVw CmZ1bmN0aW9ucyBhbmQgcG9pbnRpbmcgb3V0IHRoYXQgd2UncmUgYWN0dWFsbHkgdXNpbmcgdGhl IFdhaXQtRGllCmFsZ29yaXRobS4KCkNjOiBJbmdvIE1vbG5hciA8bWluZ29AcmVkaGF0LmNvbT4K Q2M6IEpvbmF0aGFuIENvcmJldCA8Y29yYmV0QGx3bi5uZXQ+CkNjOiBHdXN0YXZvIFBhZG92YW4g PGd1c3Rhdm9AcGFkb3Zhbi5vcmc+CkNjOiBNYWFydGVuIExhbmtob3JzdCA8bWFhcnRlbi5sYW5r aG9yc3RAbGludXguaW50ZWwuY29tPgpDYzogU2VhbiBQYXVsIDxzZWFucGF1bEBjaHJvbWl1bS5v cmc+CkNjOiBEYXZpZCBBaXJsaWUgPGFpcmxpZWRAbGludXguaWU+CkNjOiBEYXZpZGxvaHIgQnVl c28gPGRhdmVAc3Rnb2xhYnMubmV0PgpDYzogIlBhdWwgRS4gTWNLZW5uZXkiIDxwYXVsbWNrQGxp bnV4LnZuZXQuaWJtLmNvbT4KQ2M6IEpvc2ggVHJpcGxldHQgPGpvc2hAam9zaHRyaXBsZXR0Lm9y Zz4KQ2M6IFRob21hcyBHbGVpeG5lciA8dGdseEBsaW51dHJvbml4LmRlPgpDYzogS2F0ZSBTdGV3 YXJ0IDxrc3Rld2FydEBsaW51eGZvdW5kYXRpb24ub3JnPgpDYzogUGhpbGlwcGUgT21icmVkYW5u ZSA8cG9tYnJlZGFubmVAbmV4Yi5jb20+CkNjOiBHcmVnIEtyb2FoLUhhcnRtYW4gPGdyZWdraEBs aW51eGZvdW5kYXRpb24ub3JnPgpDYzogbGludXgtZG9jQHZnZXIua2VybmVsLm9yZwpDYzogbGlu dXgtbWVkaWFAdmdlci5rZXJuZWwub3JnCkNjOiBsaW5hcm8tbW0tc2lnQGxpc3RzLmxpbmFyby5v cmcKQ28tYXV0aG9yZWQtYnk6IFRob21hcyBIZWxsc3Ryb20gPHRoZWxsc3Ryb21Adm13YXJlLmNv bT4KU2lnbmVkLW9mZi1ieTogVGhvbWFzIEhlbGxzdHJvbSA8dGhlbGxzdHJvbUB2bXdhcmUuY29t PgotLS0KIERvY3VtZW50YXRpb24vbG9ja2luZy93dy1tdXRleC1kZXNpZ24udHh0IHwgIDEyICst CiBpbmNsdWRlL2xpbnV4L3d3X211dGV4LmggICAgICAgICAgICAgICAgICB8ICAyOCArKy0tLQog a2VybmVsL2xvY2tpbmcvbXV0ZXguYyAgICAgICAgICAgICAgICAgICAgfCAyMDIgKysrKysrKysr KysrKysrKysrLS0tLS0tLS0tLS0tCiAzIGZpbGVzIGNoYW5nZWQsIDE0NSBpbnNlcnRpb25zKCsp LCA5NyBkZWxldGlvbnMoLSkKCmRpZmYgLS1naXQgYS9Eb2N1bWVudGF0aW9uL2xvY2tpbmcvd3ct bXV0ZXgtZGVzaWduLnR4dCBiL0RvY3VtZW50YXRpb24vbG9ja2luZy93dy1tdXRleC1kZXNpZ24u dHh0CmluZGV4IDM0YzNhMWI1MGI5YS4uMmZkN2YyYTJhZjIxIDEwMDY0NAotLS0gYS9Eb2N1bWVu dGF0aW9uL2xvY2tpbmcvd3ctbXV0ZXgtZGVzaWduLnR4dAorKysgYi9Eb2N1bWVudGF0aW9uL2xv Y2tpbmcvd3ctbXV0ZXgtZGVzaWduLnR4dApAQCAtMzIsMTAgKzMyLDEwIEBAIHRoZSBvbGRlc3Qg dGFzaykgd2lucywgYW5kIHRoZSBvbmUgd2l0aCB0aGUgaGlnaGVyIHJlc2VydmF0aW9uIGlkIChp LmUuIHRoZQogeW91bmdlciB0YXNrKSB1bmxvY2tzIGFsbCBvZiB0aGUgYnVmZmVycyB0aGF0IGl0 IGhhcyBhbHJlYWR5IGxvY2tlZCwgYW5kIHRoZW4KIHRyaWVzIGFnYWluLgogCi1JbiB0aGUgUkRC TVMgbGl0ZXJhdHVyZSB0aGlzIGRlYWRsb2NrIGhhbmRsaW5nIGFwcHJvYWNoIGlzIGNhbGxlZCB3 YWl0L3dvdW5kOgorSW4gdGhlIFJEQk1TIGxpdGVyYXR1cmUgdGhpcyBkZWFkbG9jayBoYW5kbGlu ZyBhcHByb2FjaCBpcyBjYWxsZWQgd2FpdC9kaWU6CiBUaGUgb2xkZXIgdGFza3Mgd2FpdHMgdW50 aWwgaXQgY2FuIGFjcXVpcmUgdGhlIGNvbnRlbmRlZCBsb2NrLiBUaGUgeW91bmdlciB0YXNrcwog bmVlZHMgdG8gYmFjayBvZmYgYW5kIGRyb3AgYWxsIHRoZSBsb2NrcyBpdCBpcyBjdXJyZW50bHkg aG9sZGluZywgaS5lLiB0aGUKLXlvdW5nZXIgdGFzayBpcyB3b3VuZGVkLgoreW91bmdlciB0YXNr IGRpZXMuCiAKIENvbmNlcHRzCiAtLS0tLS0tLQpAQCAtNTYsOSArNTYsOSBAQCBGdXJ0aGVybW9y ZSB0aGVyZSBhcmUgdGhyZWUgZGlmZmVyZW50IGNsYXNzIG9mIHcvdyBsb2NrIGFjcXVpcmUgZnVu Y3Rpb25zOgogCiAqIE5vcm1hbCBsb2NrIGFjcXVpc2l0aW9uIHdpdGggYSBjb250ZXh0LCB1c2lu ZyB3d19tdXRleF9sb2NrLgogCi0qIFNsb3dwYXRoIGxvY2sgYWNxdWlzaXRpb24gb24gdGhlIGNv bnRlbmRpbmcgbG9jaywgdXNlZCBieSB0aGUgd291bmRlZCB0YXNrCi0gIGFmdGVyIGhhdmluZyBk cm9wcGVkIGFsbCBhbHJlYWR5IGFjcXVpcmVkIGxvY2tzLiBUaGVzZSBmdW5jdGlvbnMgaGF2ZSB0 aGUKLSAgX3Nsb3cgcG9zdGZpeC4KKyogU2xvd3BhdGggbG9jayBhY3F1aXNpdGlvbiBvbiB0aGUg Y29udGVuZGluZyBsb2NrLCB1c2VkIGJ5IHRoZSB0YXNrIHRoYXQganVzdAorICBraWxsZWQgaXRz IHRyYW5zYWN0aW9uIGFmdGVyIGhhdmluZyBkcm9wcGVkIGFsbCBhbHJlYWR5IGFjcXVpcmVkIGxv Y2tzLgorICBUaGVzZSBmdW5jdGlvbnMgaGF2ZSB0aGUgX3Nsb3cgcG9zdGZpeC4KIAogICBGcm9t IGEgc2ltcGxlIHNlbWFudGljcyBwb2ludC1vZi12aWV3IHRoZSBfc2xvdyBmdW5jdGlvbnMgYXJl IG5vdCBzdHJpY3RseQogICByZXF1aXJlZCwgc2luY2Ugc2ltcGx5IGNhbGxpbmcgdGhlIG5vcm1h bCB3d19tdXRleF9sb2NrIGZ1bmN0aW9ucyBvbiB0aGUKQEAgLTIyMCw3ICsyMjAsNyBAQCBtdXRl eGVzIGFyZSBhIG5hdHVyYWwgZml0IGZvciBzdWNoIGEgY2FzZSBmb3IgdHdvIHJlYXNvbnM6CiAK IE5vdGUgdGhhdCB0aGlzIGFwcHJvYWNoIGRpZmZlcnMgaW4gdHdvIGltcG9ydGFudCB3YXlzIGZy b20gdGhlIGFib3ZlIG1ldGhvZHM6CiAtIFNpbmNlIHRoZSBsaXN0IG9mIG9iamVjdHMgaXMgZHlu YW1pY2FsbHkgY29uc3RydWN0ZWQgKGFuZCBtaWdodCB2ZXJ5IHdlbGwgYmUKLSAgZGlmZmVyZW50 IHdoZW4gcmV0cnlpbmcgZHVlIHRvIGhpdHRpbmcgdGhlIC1FREVBRExLIHdvdW5kIGNvbmRpdGlv bikgdGhlcmUncworICBkaWZmZXJlbnQgd2hlbiByZXRyeWluZyBkdWUgdG8gaGl0dGluZyB0aGUg LUVERUFETEsgZGllIGNvbmRpdGlvbikgdGhlcmUncwogICBubyBuZWVkIHRvIGtlZXAgYW55IG9i amVjdCBvbiBhIHBlcnNpc3RlbnQgbGlzdCB3aGVuIGl0J3Mgbm90IGxvY2tlZC4gV2UgY2FuCiAg IHRoZXJlZm9yZSBtb3ZlIHRoZSBsaXN0X2hlYWQgaW50byB0aGUgb2JqZWN0IGl0c2VsZi4KIC0g T24gdGhlIG90aGVyIGhhbmQgdGhlIGR5bmFtaWMgb2JqZWN0IGxpc3QgY29uc3RydWN0aW9uIGFs c28gbWVhbnMgdGhhdCB0aGUgLUVBTFJFQURZIHJldHVybgpkaWZmIC0tZ2l0IGEvaW5jbHVkZS9s aW51eC93d19tdXRleC5oIGIvaW5jbHVkZS9saW51eC93d19tdXRleC5oCmluZGV4IDM5ZmRhMTk1 YmY3OC4uZjgyZmNlMjIyOWM4IDEwMDY0NAotLS0gYS9pbmNsdWRlL2xpbnV4L3d3X211dGV4LmgK KysrIGIvaW5jbHVkZS9saW51eC93d19tdXRleC5oCkBAIC02LDcgKzYsNyBAQAogICoKICAqICBD b3B5cmlnaHQgKEMpIDIwMDQsIDIwMDUsIDIwMDYgUmVkIEhhdCwgSW5jLiwgSW5nbyBNb2xuYXIg PG1pbmdvQHJlZGhhdC5jb20+CiAgKgotICogV291bmQvd2FpdCBpbXBsZW1lbnRhdGlvbjoKKyAq IFdhaXQvRGllIGltcGxlbWVudGF0aW9uOgogICogIENvcHlyaWdodCAoQykgMjAxMyBDYW5vbmlj YWwgTHRkLgogICoKICAqIFRoaXMgZmlsZSBjb250YWlucyB0aGUgbWFpbiBkYXRhIHN0cnVjdHVy ZSBhbmQgQVBJIGRlZmluaXRpb25zLgpAQCAtMjgsOSArMjgsOSBAQCBzdHJ1Y3Qgd3dfY2xhc3Mg ewogc3RydWN0IHd3X2FjcXVpcmVfY3R4IHsKIAlzdHJ1Y3QgdGFza19zdHJ1Y3QgKnRhc2s7CiAJ dW5zaWduZWQgbG9uZyBzdGFtcDsKLQl1bnNpZ25lZCBhY3F1aXJlZDsKKwl1bnNpZ25lZCBpbnQg YWNxdWlyZWQ7CiAjaWZkZWYgQ09ORklHX0RFQlVHX01VVEVYRVMKLQl1bnNpZ25lZCBkb25lX2Fj cXVpcmU7CisJdW5zaWduZWQgaW50IGRvbmVfYWNxdWlyZTsKIAlzdHJ1Y3Qgd3dfY2xhc3MgKnd3 X2NsYXNzOwogCXN0cnVjdCB3d19tdXRleCAqY29udGVuZGluZ19sb2NrOwogI2VuZGlmCkBAIC0z OCw4ICszOCw4IEBAIHN0cnVjdCB3d19hY3F1aXJlX2N0eCB7CiAJc3RydWN0IGxvY2tkZXBfbWFw IGRlcF9tYXA7CiAjZW5kaWYKICNpZmRlZiBDT05GSUdfREVCVUdfV1dfTVVURVhfU0xPV1BBVEgK LQl1bnNpZ25lZCBkZWFkbG9ja19pbmplY3RfaW50ZXJ2YWw7Ci0JdW5zaWduZWQgZGVhZGxvY2tf aW5qZWN0X2NvdW50ZG93bjsKKwl1bnNpZ25lZCBpbnQgZGVhZGxvY2tfaW5qZWN0X2ludGVydmFs OworCXVuc2lnbmVkIGludCBkZWFkbG9ja19pbmplY3RfY291bnRkb3duOwogI2VuZGlmCiB9Owog CkBAIC0xMDIsNyArMTAyLDcgQEAgc3RhdGljIGlubGluZSB2b2lkIHd3X211dGV4X2luaXQoc3Ry dWN0IHd3X211dGV4ICpsb2NrLAogICoKICAqIENvbnRleHQtYmFzZWQgdy93IG11dGV4IGFjcXVp cmluZyBjYW4gYmUgZG9uZSBpbiBhbnkgb3JkZXIgd2hhdHNvZXZlciB3aXRoaW4KICAqIGEgZ2l2 ZW4gbG9jayBjbGFzcy4gRGVhZGxvY2tzIHdpbGwgYmUgZGV0ZWN0ZWQgYW5kIGhhbmRsZWQgd2l0 aCB0aGUKLSAqIHdhaXQvd291bmQgbG9naWMuCisgKiB3YWl0L2RpZSBsb2dpYy4KICAqCiAgKiBN aXhpbmcgb2YgY29udGV4dC1iYXNlZCB3L3cgbXV0ZXggYWNxdWlyaW5nIGFuZCBzaW5nbGUgdy93 IG11dGV4IGxvY2tpbmcgY2FuCiAgKiByZXN1bHQgaW4gdW5kZXRlY3RlZCBkZWFkbG9ja3MgYW5k IGlzIHNvIGZvcmJpZGRlbi4gTWl4aW5nIGRpZmZlcmVudCBjb250ZXh0cwpAQCAtMTk1LDEzICsx OTUsMTMgQEAgc3RhdGljIGlubGluZSB2b2lkIHd3X2FjcXVpcmVfZmluaShzdHJ1Y3Qgd3dfYWNx dWlyZV9jdHggKmN0eCkKICAqIExvY2sgdGhlIHcvdyBtdXRleCBleGNsdXNpdmVseSBmb3IgdGhp cyB0YXNrLgogICoKICAqIERlYWRsb2NrcyB3aXRoaW4gYSBnaXZlbiB3L3cgY2xhc3Mgb2YgbG9j a3MgYXJlIGRldGVjdGVkIGFuZCBoYW5kbGVkIHdpdGggdGhlCi0gKiB3YWl0L3dvdW5kIGFsZ29y aXRobS4gSWYgdGhlIGxvY2sgaXNuJ3QgaW1tZWRpYXRlbHkgYXZhaWFibGUgdGhpcyBmdW5jdGlv bgorICogd2FpdC9kaWUgYWxnb3JpdGhtLiBJZiB0aGUgbG9jayBpc24ndCBpbW1lZGlhdGVseSBh dmFpbGFibGUgdGhpcyBmdW5jdGlvbgogICogd2lsbCBlaXRoZXIgc2xlZXAgdW50aWwgaXQgaXMg KHdhaXQgY2FzZSkuIE9yIGl0IHNlbGVjdHMgdGhlIGN1cnJlbnQgY29udGV4dAotICogZm9yIGJh Y2tpbmcgb2ZmIGJ5IHJldHVybmluZyAtRURFQURMSyAod291bmQgY2FzZSkuIFRyeWluZyB0byBh Y3F1aXJlIHRoZQorICogZm9yIGJhY2tpbmcgb2ZmIGJ5IHJldHVybmluZyAtRURFQURMSyAoZGll IGNhc2UpLiBUcnlpbmcgdG8gYWNxdWlyZSB0aGUKICAqIHNhbWUgbG9jayB3aXRoIHRoZSBzYW1l IGNvbnRleHQgdHdpY2UgaXMgYWxzbyBkZXRlY3RlZCBhbmQgc2lnbmFsbGVkIGJ5CiAgKiByZXR1 cm5pbmcgLUVBTFJFQURZLiBSZXR1cm5zIDAgaWYgdGhlIG11dGV4IHdhcyBzdWNjZXNzZnVsbHkg YWNxdWlyZWQuCiAgKgotICogSW4gdGhlIHdvdW5kIGNhc2UgdGhlIGNhbGxlciBtdXN0IHJlbGVh c2UgYWxsIGN1cnJlbnRseSBoZWxkIHcvdyBtdXRleGVzIGZvcgorICogSW4gdGhlIGRpZSBjYXNl IHRoZSBjYWxsZXIgbXVzdCByZWxlYXNlIGFsbCBjdXJyZW50bHkgaGVsZCB3L3cgbXV0ZXhlcyBm b3IKICAqIHRoZSBnaXZlbiBjb250ZXh0IGFuZCB0aGVuIHdhaXQgZm9yIHRoaXMgY29udGVuZGlu ZyBsb2NrIHRvIGJlIGF2YWlsYWJsZSBieQogICogY2FsbGluZyB3d19tdXRleF9sb2NrX3Nsb3cu IEFsdGVybmF0aXZlbHkgY2FsbGVycyBjYW4gb3B0IHRvIG5vdCBhY3F1aXJlIHRoaXMKICAqIGxv Y2sgYW5kIHByb2NlZWQgd2l0aCB0cnlpbmcgdG8gYWNxdWlyZSBmdXJ0aGVyIHcvdyBtdXRleGVz IChlLmcuIHdoZW4KQEAgLTIyNiwxNCArMjI2LDE0IEBAIGV4dGVybiBpbnQgLyogX19tdXN0X2No ZWNrICovIHd3X211dGV4X2xvY2soc3RydWN0IHd3X211dGV4ICpsb2NrLCBzdHJ1Y3Qgd3dfYWNx CiAgKiBMb2NrIHRoZSB3L3cgbXV0ZXggZXhjbHVzaXZlbHkgZm9yIHRoaXMgdGFzay4KICAqCiAg KiBEZWFkbG9ja3Mgd2l0aGluIGEgZ2l2ZW4gdy93IGNsYXNzIG9mIGxvY2tzIGFyZSBkZXRlY3Rl ZCBhbmQgaGFuZGxlZCB3aXRoIHRoZQotICogd2FpdC93b3VuZCBhbGdvcml0aG0uIElmIHRoZSBs b2NrIGlzbid0IGltbWVkaWF0ZWx5IGF2YWlhYmxlIHRoaXMgZnVuY3Rpb24KKyAqIHdhaXQvZGll IGFsZ29yaXRobS4gSWYgdGhlIGxvY2sgaXNuJ3QgaW1tZWRpYXRlbHkgYXZhaWxhYmxlIHRoaXMg ZnVuY3Rpb24KICAqIHdpbGwgZWl0aGVyIHNsZWVwIHVudGlsIGl0IGlzICh3YWl0IGNhc2UpLiBP ciBpdCBzZWxlY3RzIHRoZSBjdXJyZW50IGNvbnRleHQKLSAqIGZvciBiYWNraW5nIG9mZiBieSBy ZXR1cm5pbmcgLUVERUFETEsgKHdvdW5kIGNhc2UpLiBUcnlpbmcgdG8gYWNxdWlyZSB0aGUKKyAq IGZvciBiYWNraW5nIG9mZiBieSByZXR1cm5pbmcgLUVERUFETEsgKGRpZSBjYXNlKS4gVHJ5aW5n IHRvIGFjcXVpcmUgdGhlCiAgKiBzYW1lIGxvY2sgd2l0aCB0aGUgc2FtZSBjb250ZXh0IHR3aWNl IGlzIGFsc28gZGV0ZWN0ZWQgYW5kIHNpZ25hbGxlZCBieQogICogcmV0dXJuaW5nIC1FQUxSRUFE WS4gUmV0dXJucyAwIGlmIHRoZSBtdXRleCB3YXMgc3VjY2Vzc2Z1bGx5IGFjcXVpcmVkLiBJZiBh CiAgKiBzaWduYWwgYXJyaXZlcyB3aGlsZSB3YWl0aW5nIGZvciB0aGUgbG9jayB0aGVuIHRoaXMg ZnVuY3Rpb24gcmV0dXJucyAtRUlOVFIuCiAgKgotICogSW4gdGhlIHdvdW5kIGNhc2UgdGhlIGNh bGxlciBtdXN0IHJlbGVhc2UgYWxsIGN1cnJlbnRseSBoZWxkIHcvdyBtdXRleGVzIGZvcgorICog SW4gdGhlIGRpZSBjYXNlIHRoZSBjYWxsZXIgbXVzdCByZWxlYXNlIGFsbCBjdXJyZW50bHkgaGVs ZCB3L3cgbXV0ZXhlcyBmb3IKICAqIHRoZSBnaXZlbiBjb250ZXh0IGFuZCB0aGVuIHdhaXQgZm9y IHRoaXMgY29udGVuZGluZyBsb2NrIHRvIGJlIGF2YWlsYWJsZSBieQogICogY2FsbGluZyB3d19t dXRleF9sb2NrX3Nsb3dfaW50ZXJydXB0aWJsZS4gQWx0ZXJuYXRpdmVseSBjYWxsZXJzIGNhbiBv cHQgdG8KICAqIG5vdCBhY3F1aXJlIHRoaXMgbG9jayBhbmQgcHJvY2VlZCB3aXRoIHRyeWluZyB0 byBhY3F1aXJlIGZ1cnRoZXIgdy93IG11dGV4ZXMKQEAgLTI1Niw3ICsyNTYsNyBAQCBleHRlcm4g aW50IF9fbXVzdF9jaGVjayB3d19tdXRleF9sb2NrX2ludGVycnVwdGlibGUoc3RydWN0IHd3X211 dGV4ICpsb2NrLAogICogQGxvY2s6IHRoZSBtdXRleCB0byBiZSBhY3F1aXJlZAogICogQGN0eDog dy93IGFjcXVpcmUgY29udGV4dAogICoKLSAqIEFjcXVpcmVzIGEgdy93IG11dGV4IHdpdGggdGhl IGdpdmVuIGNvbnRleHQgYWZ0ZXIgYSB3b3VuZCBjYXNlLiBUaGlzIGZ1bmN0aW9uCisgKiBBY3F1 aXJlcyBhIHcvdyBtdXRleCB3aXRoIHRoZSBnaXZlbiBjb250ZXh0IGFmdGVyIGEgZGllIGNhc2Uu IFRoaXMgZnVuY3Rpb24KICAqIHdpbGwgc2xlZXAgdW50aWwgdGhlIGxvY2sgYmVjb21lcyBhdmFp bGFibGUuCiAgKgogICogVGhlIGNhbGxlciBtdXN0IGhhdmUgcmVsZWFzZWQgYWxsIHcvdyBtdXRl eGVzIGFscmVhZHkgYWNxdWlyZWQgd2l0aCB0aGUKQEAgLTI5MCw3ICsyOTAsNyBAQCB3d19tdXRl eF9sb2NrX3Nsb3coc3RydWN0IHd3X211dGV4ICpsb2NrLCBzdHJ1Y3Qgd3dfYWNxdWlyZV9jdHgg KmN0eCkKICAqIEBsb2NrOiB0aGUgbXV0ZXggdG8gYmUgYWNxdWlyZWQKICAqIEBjdHg6IHcvdyBh Y3F1aXJlIGNvbnRleHQKICAqCi0gKiBBY3F1aXJlcyBhIHcvdyBtdXRleCB3aXRoIHRoZSBnaXZl biBjb250ZXh0IGFmdGVyIGEgd291bmQgY2FzZS4gVGhpcyBmdW5jdGlvbgorICogQWNxdWlyZXMg YSB3L3cgbXV0ZXggd2l0aCB0aGUgZ2l2ZW4gY29udGV4dCBhZnRlciBhIGRpZSBjYXNlLiBUaGlz IGZ1bmN0aW9uCiAgKiB3aWxsIHNsZWVwIHVudGlsIHRoZSBsb2NrIGJlY29tZXMgYXZhaWxhYmxl IGFuZCByZXR1cm5zIDAgd2hlbiB0aGUgbG9jayBoYXMKICAqIGJlZW4gYWNxdWlyZWQuIElmIGEg c2lnbmFsIGFycml2ZXMgd2hpbGUgd2FpdGluZyBmb3IgdGhlIGxvY2sgdGhlbiB0aGlzCiAgKiBm dW5jdGlvbiByZXR1cm5zIC1FSU5UUi4KZGlmZiAtLWdpdCBhL2tlcm5lbC9sb2NraW5nL211dGV4 LmMgYi9rZXJuZWwvbG9ja2luZy9tdXRleC5jCmluZGV4IDIwNDgzNTlmMzNkMi4uNDEyYjRmYzA4 MjM1IDEwMDY0NAotLS0gYS9rZXJuZWwvbG9ja2luZy9tdXRleC5jCisrKyBiL2tlcm5lbC9sb2Nr aW5nL211dGV4LmMKQEAgLTI0Myw2ICsyNDMsMTcgQEAgdm9pZCBfX3NjaGVkIG11dGV4X2xvY2so c3RydWN0IG11dGV4ICpsb2NrKQogRVhQT1JUX1NZTUJPTChtdXRleF9sb2NrKTsKICNlbmRpZgog CisvKgorICogV2FpdC1EaWU6CisgKiAgIFRoZSBuZXdlciB0cmFuc2FjdGlvbnMgYXJlIGtpbGxl ZCB3aGVuOgorICogICAgIEl0ICh0aGUgbmV3IHRyYW5zYWN0aW9uKSBtYWtlcyBhIHJlcXVlc3Qg Zm9yIGEgbG9jayBiZWluZyBoZWxkCisgKiAgICAgYnkgYW4gb2xkZXIgdHJhbnNhY3Rpb24uCisg Ki8KKworLyoKKyAqIEFzc29jaWF0ZSB0aGUgd3dfbXV0ZXggQHd3IHdpdGggdGhlIGNvbnRleHQg QHd3X2N0eCB1bmRlciB3aGljaCB3ZSBhY3F1aXJlZAorICogaXQuCisgKi8KIHN0YXRpYyBfX2Fs d2F5c19pbmxpbmUgdm9pZAogd3dfbXV0ZXhfbG9ja19hY3F1aXJlZChzdHJ1Y3Qgd3dfbXV0ZXgg Knd3LCBzdHJ1Y3Qgd3dfYWNxdWlyZV9jdHggKnd3X2N0eCkKIHsKQEAgLTI4MSwyNiArMjkyLDUz IEBAIHd3X211dGV4X2xvY2tfYWNxdWlyZWQoc3RydWN0IHd3X211dGV4ICp3dywgc3RydWN0IHd3 X2FjcXVpcmVfY3R4ICp3d19jdHgpCiAJREVCVUdfTE9DS1NfV0FSTl9PTih3d19jdHgtPnd3X2Ns YXNzICE9IHd3LT53d19jbGFzcyk7CiAjZW5kaWYKIAl3d19jdHgtPmFjcXVpcmVkKys7CisJd3ct PmN0eCA9IHd3X2N0eDsKIH0KIAorLyoKKyAqIERldGVybWluZSBpZiBjb250ZXh0IEBhIGlzICdh ZnRlcicgY29udGV4dCBAYi4gSU9XLCBAYSBpcyBhIHlvdW5nZXIKKyAqIHRyYW5zYWN0aW9uIHRo YW4gQGIgYW5kIGRlcGVuZGluZyBvbiBhbGdvcml0aG0gZWl0aGVyIG5lZWRzIHRvIHdhaXQgZm9y CisgKiBAYiBvciBkaWUuCisgKi8KIHN0YXRpYyBpbmxpbmUgYm9vbCBfX3NjaGVkCiBfX3d3X2N0 eF9zdGFtcF9hZnRlcihzdHJ1Y3Qgd3dfYWNxdWlyZV9jdHggKmEsIHN0cnVjdCB3d19hY3F1aXJl X2N0eCAqYikKIHsKLQlyZXR1cm4gYS0+c3RhbXAgLSBiLT5zdGFtcCA8PSBMT05HX01BWCAmJgot CSAgICAgICAoYS0+c3RhbXAgIT0gYi0+c3RhbXAgfHwgYSA+IGIpOworCisJcmV0dXJuIChzaWdu ZWQgbG9uZykoYS0+c3RhbXAgLSBiLT5zdGFtcCkgPiAwOworfQorCisvKgorICogV2FpdC1EaWU7 IHdha2UgYSB5b3VuZ2VyIHdhaXRlciBjb250ZXh0ICh3aGVuIGxvY2tzIGhlbGQpIHN1Y2ggdGhh dCBpdCBjYW4KKyAqIGRpZS4KKyAqCisgKiBBbW9uZyB3YWl0ZXJzIHdpdGggY29udGV4dCwgb25s eSB0aGUgZmlyc3Qgb25lIGNhbiBoYXZlIG90aGVyIGxvY2tzIGFjcXVpcmVkCisgKiBhbHJlYWR5 IChjdHgtPmFjcXVpcmVkID4gMCksIGJlY2F1c2UgX193d19tdXRleF9hZGRfd2FpdGVyKCkgYW5k CisgKiBfX3d3X211dGV4X2NoZWNrX2tpbGwoKSB3YWtlIGFueSBidXQgdGhlIGVhcmxpZXN0IGNv bnRleHQuCisgKi8KK3N0YXRpYyBib29sIF9fc2NoZWQKK19fd3dfbXV0ZXhfZGllKHN0cnVjdCBt dXRleCAqbG9jaywgc3RydWN0IG11dGV4X3dhaXRlciAqd2FpdGVyLAorCSAgICAgICBzdHJ1Y3Qg d3dfYWNxdWlyZV9jdHggKnd3X2N0eCkKK3sKKwlpZiAod2FpdGVyLT53d19jdHgtPmFjcXVpcmVk ID4gMCAmJgorCQkJX193d19jdHhfc3RhbXBfYWZ0ZXIod2FpdGVyLT53d19jdHgsIHd3X2N0eCkp IHsKKwkJZGVidWdfbXV0ZXhfd2FrZV93YWl0ZXIobG9jaywgd2FpdGVyKTsKKwkJd2FrZV91cF9w cm9jZXNzKHdhaXRlci0+dGFzayk7CisJfQorCisJcmV0dXJuIHRydWU7CiB9CiAKIC8qCi0gKiBX YWtlIHVwIGFueSB3YWl0ZXJzIHRoYXQgbWF5IGhhdmUgdG8gYmFjayBvZmYgd2hlbiB0aGUgbG9j ayBpcyBoZWxkIGJ5IHRoZQotICogZ2l2ZW4gY29udGV4dC4KKyAqIFdlIGp1c3QgYWNxdWlyZWQg QGxvY2sgdW5kZXIgQHd3X2N0eCwgaWYgdGhlcmUgYXJlIGxhdGVyIGNvbnRleHRzIHdhaXRpbmcK KyAqIGJlaGluZCB1cyBvbiB0aGUgd2FpdC1saXN0LCBjaGVjayBpZiB0aGV5IG5lZWQgdG8gZGll LgogICoKLSAqIER1ZSB0byB0aGUgaW52YXJpYW50cyBvbiB0aGUgd2FpdCBsaXN0LCB0aGlzIGNh biBvbmx5IGFmZmVjdCB0aGUgZmlyc3QKLSAqIHdhaXRlciB3aXRoIGEgY29udGV4dC4KKyAqIFNl ZSBfX3d3X211dGV4X2FkZF93YWl0ZXIoKSBmb3IgdGhlIGxpc3Qtb3JkZXIgY29uc3RydWN0aW9u OyBiYXNpY2FsbHkgdGhlCisgKiBsaXN0IGlzIG9yZGVyZWQgYnkgc3RhbXAsIHNtYWxsZXN0IChv bGRlc3QpIGZpcnN0LgogICoKICAqIFRoZSBjdXJyZW50IHRhc2sgbXVzdCBub3QgYmUgb24gdGhl IHdhaXQgbGlzdC4KICAqLwogc3RhdGljIHZvaWQgX19zY2hlZAotX193d19tdXRleF93YWtldXBf Zm9yX2JhY2tvZmYoc3RydWN0IG11dGV4ICpsb2NrLCBzdHJ1Y3Qgd3dfYWNxdWlyZV9jdHggKnd3 X2N0eCkKK19fd3dfbXV0ZXhfY2hlY2tfd2FpdGVycyhzdHJ1Y3QgbXV0ZXggKmxvY2ssIHN0cnVj dCB3d19hY3F1aXJlX2N0eCAqd3dfY3R4KQogewogCXN0cnVjdCBtdXRleF93YWl0ZXIgKmN1cjsK IApAQCAtMzEwLDMwICszNDgsMjMgQEAgX193d19tdXRleF93YWtldXBfZm9yX2JhY2tvZmYoc3Ry dWN0IG11dGV4ICpsb2NrLCBzdHJ1Y3Qgd3dfYWNxdWlyZV9jdHggKnd3X2N0eCkKIAkJaWYgKCFj dXItPnd3X2N0eCkKIAkJCWNvbnRpbnVlOwogCi0JCWlmIChjdXItPnd3X2N0eC0+YWNxdWlyZWQg PiAwICYmCi0JCSAgICBfX3d3X2N0eF9zdGFtcF9hZnRlcihjdXItPnd3X2N0eCwgd3dfY3R4KSkg ewotCQkJZGVidWdfbXV0ZXhfd2FrZV93YWl0ZXIobG9jaywgY3VyKTsKLQkJCXdha2VfdXBfcHJv Y2VzcyhjdXItPnRhc2spOwotCQl9Ci0KLQkJYnJlYWs7CisJCWlmIChfX3d3X211dGV4X2RpZShs b2NrLCBjdXIsIHd3X2N0eCkpCisJCQlicmVhazsKIAl9CiB9CiAKIC8qCi0gKiBBZnRlciBhY3F1 aXJpbmcgbG9jayB3aXRoIGZhc3RwYXRoIG9yIHdoZW4gd2UgbG9zdCBvdXQgaW4gY29udGVzdGVk Ci0gKiBzbG93cGF0aCwgc2V0IGN0eCBhbmQgd2FrZSB1cCBhbnkgd2FpdGVycyBzbyB0aGV5IGNh biByZWNoZWNrLgorICogQWZ0ZXIgYWNxdWlyaW5nIGxvY2sgd2l0aCBmYXN0cGF0aCwgd2hlcmUg d2UgZG8gbm90IGhvbGQgd2FpdF9sb2NrLCBzZXQgY3R4CisgKiBhbmQgd2FrZSB1cCBhbnkgd2Fp dGVycyBzbyB0aGV5IGNhbiByZWNoZWNrLgogICovCiBzdGF0aWMgX19hbHdheXNfaW5saW5lIHZv aWQKIHd3X211dGV4X3NldF9jb250ZXh0X2Zhc3RwYXRoKHN0cnVjdCB3d19tdXRleCAqbG9jaywg c3RydWN0IHd3X2FjcXVpcmVfY3R4ICpjdHgpCiB7CiAJd3dfbXV0ZXhfbG9ja19hY3F1aXJlZChs b2NrLCBjdHgpOwogCi0JbG9jay0+Y3R4ID0gY3R4OwotCiAJLyoKIAkgKiBUaGUgbG9jay0+Y3R4 IHVwZGF0ZSBzaG91bGQgYmUgdmlzaWJsZSBvbiBhbGwgY29yZXMgYmVmb3JlCi0JICogdGhlIGF0 b21pYyByZWFkIGlzIGRvbmUsIG90aGVyd2lzZSBjb250ZW5kZWQgd2FpdGVycyBtaWdodCBiZQor CSAqIHRoZSBXQUlURVJTIGNoZWNrIGlzIGRvbmUsIG90aGVyd2lzZSBjb250ZW5kZWQgd2FpdGVy cyBtaWdodCBiZQogCSAqIG1pc3NlZC4gVGhlIGNvbnRlbmRlZCB3YWl0ZXJzIHdpbGwgZWl0aGVy IHNlZSB3d19jdHggPT0gTlVMTAogCSAqIGFuZCBrZWVwIHNwaW5uaW5nLCBvciBpdCB3aWxsIGFj cXVpcmUgd2FpdF9sb2NrLCBhZGQgaXRzZWxmCiAJICogdG8gd2FpdGVyIGxpc3QgYW5kIHNsZWVw LgpAQCAtMzQ3LDI5ICszNzgsMTQgQEAgd3dfbXV0ZXhfc2V0X2NvbnRleHRfZmFzdHBhdGgoc3Ry dWN0IHd3X211dGV4ICpsb2NrLCBzdHJ1Y3Qgd3dfYWNxdWlyZV9jdHggKmN0eCkKIAkJcmV0dXJu OwogCiAJLyoKLQkgKiBVaCBvaCwgd2UgcmFjZWQgaW4gZmFzdHBhdGgsIHdha2UgdXAgZXZlcnlv bmUgaW4gdGhpcyBjYXNlLAotCSAqIHNvIHRoZXkgY2FuIHNlZSB0aGUgbmV3IGxvY2stPmN0eC4K KwkgKiBVaCBvaCwgd2UgcmFjZWQgaW4gZmFzdHBhdGgsIGNoZWNrIGlmIGFueSBvZiB0aGUgd2Fp dGVycyBuZWVkIHRvCisJICogZGllLgogCSAqLwogCXNwaW5fbG9jaygmbG9jay0+YmFzZS53YWl0 X2xvY2spOwotCV9fd3dfbXV0ZXhfd2FrZXVwX2Zvcl9iYWNrb2ZmKCZsb2NrLT5iYXNlLCBjdHgp OworCV9fd3dfbXV0ZXhfY2hlY2tfd2FpdGVycygmbG9jay0+YmFzZSwgY3R4KTsKIAlzcGluX3Vu bG9jaygmbG9jay0+YmFzZS53YWl0X2xvY2spOwogfQogCi0vKgotICogQWZ0ZXIgYWNxdWlyaW5n IGxvY2sgaW4gdGhlIHNsb3dwYXRoIHNldCBjdHguCi0gKgotICogVW5saWtlIGZvciB0aGUgZmFz dCBwYXRoLCB0aGUgY2FsbGVyIGVuc3VyZXMgdGhhdCB3YWl0ZXJzIGFyZSB3b2tlbiB1cCB3aGVy ZQotICogbmVjZXNzYXJ5LgotICoKLSAqIENhbGxlcnMgbXVzdCBob2xkIHRoZSBtdXRleCB3YWl0 X2xvY2suCi0gKi8KLXN0YXRpYyBfX2Fsd2F5c19pbmxpbmUgdm9pZAotd3dfbXV0ZXhfc2V0X2Nv bnRleHRfc2xvd3BhdGgoc3RydWN0IHd3X211dGV4ICpsb2NrLCBzdHJ1Y3Qgd3dfYWNxdWlyZV9j dHggKmN0eCkKLXsKLQl3d19tdXRleF9sb2NrX2FjcXVpcmVkKGxvY2ssIGN0eCk7Ci0JbG9jay0+ Y3R4ID0gY3R4OwotfQotCiAjaWZkZWYgQ09ORklHX01VVEVYX1NQSU5fT05fT1dORVIKIAogc3Rh dGljIGlubGluZQpAQCAtNjQ1LDM3ICs2NjEsNzMgQEAgdm9pZCBfX3NjaGVkIHd3X211dGV4X3Vu bG9jayhzdHJ1Y3Qgd3dfbXV0ZXggKmxvY2spCiB9CiBFWFBPUlRfU1lNQk9MKHd3X211dGV4X3Vu bG9jayk7CiAKKworc3RhdGljIF9fYWx3YXlzX2lubGluZSBpbnQgX19zY2hlZAorX193d19tdXRl eF9raWxsKHN0cnVjdCBtdXRleCAqbG9jaywgc3RydWN0IHd3X2FjcXVpcmVfY3R4ICp3d19jdHgp Cit7CisJaWYgKHd3X2N0eC0+YWNxdWlyZWQgPiAwKSB7CisjaWZkZWYgQ09ORklHX0RFQlVHX01V VEVYRVMKKwkJc3RydWN0IHd3X211dGV4ICp3dzsKKworCQl3dyA9IGNvbnRhaW5lcl9vZihsb2Nr LCBzdHJ1Y3Qgd3dfbXV0ZXgsIGJhc2UpOworCQlERUJVR19MT0NLU19XQVJOX09OKHd3X2N0eC0+ Y29udGVuZGluZ19sb2NrKTsKKwkJd3dfY3R4LT5jb250ZW5kaW5nX2xvY2sgPSB3dzsKKyNlbmRp ZgorCQlyZXR1cm4gLUVERUFETEs7CisJfQorCisJcmV0dXJuIDA7Cit9CisKKworLyoKKyAqIENo ZWNrIHdoZXRoZXIgd2UgbmVlZCB0byBraWxsIHRoZSB0cmFuc2FjdGlvbiBmb3IgdGhlIGN1cnJl bnQgbG9jayBhY3F1aXJlLgorICoKKyAqIFdhaXQtRGllOiBJZiB3ZSdyZSB0cnlpbmcgdG8gYWNx dWlyZSBhIGxvY2sgYWxyZWFkeSBoZWxkIGJ5IGFuIG9sZGVyCisgKiAgICAgICAgICAgY29udGV4 dCwga2lsbCBvdXJzZWx2ZXMuCisgKgorICogU2luY2UgX193d19tdXRleF9hZGRfd2FpdGVyKCkg b3JkZXJzIHRoZSB3YWl0LWxpc3Qgb24gc3RhbXAsIHdlIG9ubHkgaGF2ZSB0bworICogbG9vayBh dCB3YWl0ZXJzIGJlZm9yZSB1cyBpbiB0aGUgd2FpdC1saXN0LgorICovCiBzdGF0aWMgaW5saW5l IGludCBfX3NjaGVkCi1fX3d3X211dGV4X2xvY2tfY2hlY2tfc3RhbXAoc3RydWN0IG11dGV4ICps b2NrLCBzdHJ1Y3QgbXV0ZXhfd2FpdGVyICp3YWl0ZXIsCi0JCQkgICAgc3RydWN0IHd3X2FjcXVp cmVfY3R4ICpjdHgpCitfX3d3X211dGV4X2NoZWNrX2tpbGwoc3RydWN0IG11dGV4ICpsb2NrLCBz dHJ1Y3QgbXV0ZXhfd2FpdGVyICp3YWl0ZXIsCisJCSAgICAgIHN0cnVjdCB3d19hY3F1aXJlX2N0 eCAqY3R4KQogewogCXN0cnVjdCB3d19tdXRleCAqd3cgPSBjb250YWluZXJfb2YobG9jaywgc3Ry dWN0IHd3X211dGV4LCBiYXNlKTsKIAlzdHJ1Y3Qgd3dfYWNxdWlyZV9jdHggKmhvbGRfY3R4ID0g UkVBRF9PTkNFKHd3LT5jdHgpOwogCXN0cnVjdCBtdXRleF93YWl0ZXIgKmN1cjsKIAorCWlmIChj dHgtPmFjcXVpcmVkID09IDApCisJCXJldHVybiAwOworCiAJaWYgKGhvbGRfY3R4ICYmIF9fd3df Y3R4X3N0YW1wX2FmdGVyKGN0eCwgaG9sZF9jdHgpKQotCQlnb3RvIGRlYWRsb2NrOworCQlyZXR1 cm4gX193d19tdXRleF9raWxsKGxvY2ssIGN0eCk7CiAKIAkvKgogCSAqIElmIHRoZXJlIGlzIGEg d2FpdGVyIGluIGZyb250IG9mIHVzIHRoYXQgaGFzIGEgY29udGV4dCwgdGhlbiBpdHMKLQkgKiBz dGFtcCBpcyBlYXJsaWVyIHRoYW4gb3VycyBhbmQgd2UgbXVzdCBiYWNrIG9mZi4KKwkgKiBzdGFt cCBpcyBlYXJsaWVyIHRoYW4gb3VycyBhbmQgd2UgbXVzdCBraWxsIG91cnNlbGYuCiAJICovCiAJ Y3VyID0gd2FpdGVyOwogCWxpc3RfZm9yX2VhY2hfZW50cnlfY29udGludWVfcmV2ZXJzZShjdXIs ICZsb2NrLT53YWl0X2xpc3QsIGxpc3QpIHsKLQkJaWYgKGN1ci0+d3dfY3R4KQotCQkJZ290byBk ZWFkbG9jazsKKwkJaWYgKCFjdXItPnd3X2N0eCkKKwkJCWNvbnRpbnVlOworCisJCXJldHVybiBf X3d3X211dGV4X2tpbGwobG9jaywgY3R4KTsKIAl9CiAKIAlyZXR1cm4gMDsKLQotZGVhZGxvY2s6 Ci0jaWZkZWYgQ09ORklHX0RFQlVHX01VVEVYRVMKLQlERUJVR19MT0NLU19XQVJOX09OKGN0eC0+ Y29udGVuZGluZ19sb2NrKTsKLQljdHgtPmNvbnRlbmRpbmdfbG9jayA9IHd3OwotI2VuZGlmCi0J cmV0dXJuIC1FREVBRExLOwogfQogCisvKgorICogQWRkIEB3YWl0ZXIgdG8gdGhlIHdhaXQtbGlz dCwga2VlcCB0aGUgd2FpdC1saXN0IG9yZGVyZWQgYnkgc3RhbXAsIHNtYWxsZXN0CisgKiBmaXJz dC4gU3VjaCB0aGF0IG9sZGVyIGNvbnRleHRzIGFyZSBwcmVmZXJyZWQgdG8gYWNxdWlyZSB0aGUg bG9jayBvdmVyCisgKiB5b3VuZ2VyIGNvbnRleHRzLgorICoKKyAqIFdhaXRlcnMgd2l0aG91dCBj b250ZXh0IGFyZSBpbnRlcnNwZXJzZWQgaW4gRklGTyBvcmRlci4KKyAqCisgKiBGdXJ0aGVybW9y ZSwgZm9yIFdhaXQtRGllIGtpbGwgb3Vyc2VsZiBpbW1lZGlhdGVseSB3aGVuIHBvc3NpYmxlICh0 aGVyZSBhcmUKKyAqIG9sZGVyIGNvbnRleHRzIGFscmVhZHkgd2FpdGluZykgdG8gYXZvaWQgdW5u ZWNlc3Nhcnkgd2FpdGluZy4KKyAqLwogc3RhdGljIGlubGluZSBpbnQgX19zY2hlZAogX193d19t dXRleF9hZGRfd2FpdGVyKHN0cnVjdCBtdXRleF93YWl0ZXIgKndhaXRlciwKIAkJICAgICAgc3Ry dWN0IG11dGV4ICpsb2NrLApAQCAtNjkyLDcgKzc0NCw3IEBAIF9fd3dfbXV0ZXhfYWRkX3dhaXRl cihzdHJ1Y3QgbXV0ZXhfd2FpdGVyICp3YWl0ZXIsCiAJLyoKIAkgKiBBZGQgdGhlIHdhaXRlciBi ZWZvcmUgdGhlIGZpcnN0IHdhaXRlciB3aXRoIGEgaGlnaGVyIHN0YW1wLgogCSAqIFdhaXRlcnMg d2l0aG91dCBhIGNvbnRleHQgYXJlIHNraXBwZWQgdG8gYXZvaWQgc3RhcnZpbmcKLQkgKiB0aGVt LgorCSAqIHRoZW0uIFdhaXQtRGllIHdhaXRlcnMgbWF5IGRpZSBoZXJlLgogCSAqLwogCXBvcyA9 ICZsb2NrLT53YWl0X2xpc3Q7CiAJbGlzdF9mb3JfZWFjaF9lbnRyeV9yZXZlcnNlKGN1ciwgJmxv Y2stPndhaXRfbGlzdCwgbGlzdCkgewpAQCAtNzAwLDM0ICs3NTIsMjcgQEAgX193d19tdXRleF9h ZGRfd2FpdGVyKHN0cnVjdCBtdXRleF93YWl0ZXIgKndhaXRlciwKIAkJCWNvbnRpbnVlOwogCiAJ CWlmIChfX3d3X2N0eF9zdGFtcF9hZnRlcih3d19jdHgsIGN1ci0+d3dfY3R4KSkgewotCQkJLyog QmFjayBvZmYgaW1tZWRpYXRlbHkgaWYgbmVjZXNzYXJ5LiAqLwotCQkJaWYgKHd3X2N0eC0+YWNx dWlyZWQgPiAwKSB7Ci0jaWZkZWYgQ09ORklHX0RFQlVHX01VVEVYRVMKLQkJCQlzdHJ1Y3Qgd3df bXV0ZXggKnd3OworCQkJLyoKKwkJCSAqIFdhaXQtRGllOiBpZiB3ZSBmaW5kIGFuIG9sZGVyIGNv bnRleHQgd2FpdGluZywgdGhlcmUKKwkJCSAqIGlzIG5vIHBvaW50IGluIHF1ZXVlaW5nIGJlaGlu ZCBpdCwgYXMgd2UnZCBoYXZlIHRvCisJCQkgKiBkaWUgdGhlIG1vbWVudCBpdCB3b3VsZCBhY3F1 aXJlIHRoZSBsb2NrLgorCQkJICovCisJCQlpbnQgcmV0ID0gX193d19tdXRleF9raWxsKGxvY2ss IHd3X2N0eCk7CiAKLQkJCQl3dyA9IGNvbnRhaW5lcl9vZihsb2NrLCBzdHJ1Y3Qgd3dfbXV0ZXgs IGJhc2UpOwotCQkJCURFQlVHX0xPQ0tTX1dBUk5fT04od3dfY3R4LT5jb250ZW5kaW5nX2xvY2sp OwotCQkJCXd3X2N0eC0+Y29udGVuZGluZ19sb2NrID0gd3c7Ci0jZW5kaWYKLQkJCQlyZXR1cm4g LUVERUFETEs7Ci0JCQl9CisJCQlpZiAocmV0KQorCQkJCXJldHVybiByZXQ7CiAKIAkJCWJyZWFr OwogCQl9CiAKIAkJcG9zID0gJmN1ci0+bGlzdDsKIAotCQkvKgotCQkgKiBXYWtlIHVwIHRoZSB3 YWl0ZXIgc28gdGhhdCBpdCBnZXRzIGEgY2hhbmNlIHRvIGJhY2sKLQkJICogb2ZmLgotCQkgKi8K LQkJaWYgKGN1ci0+d3dfY3R4LT5hY3F1aXJlZCA+IDApIHsKLQkJCWRlYnVnX211dGV4X3dha2Vf d2FpdGVyKGxvY2ssIGN1cik7Ci0JCQl3YWtlX3VwX3Byb2Nlc3MoY3VyLT50YXNrKTsKLQkJfQor CQkvKiBXYWl0LURpZTogZW5zdXJlIHlvdW5nZXIgd2FpdGVycyBkaWUuICovCisJCV9fd3dfbXV0 ZXhfZGllKGxvY2ssIGN1ciwgd3dfY3R4KTsKIAl9CiAKIAlsaXN0X2FkZF90YWlsKCZ3YWl0ZXIt Pmxpc3QsIHBvcyk7CisKIAlyZXR1cm4gMDsKIH0KIApAQCAtNzcxLDcgKzgxNiw3IEBAIF9fbXV0 ZXhfbG9ja19jb21tb24oc3RydWN0IG11dGV4ICpsb2NrLCBsb25nIHN0YXRlLCB1bnNpZ25lZCBp bnQgc3ViY2xhc3MsCiAJICovCiAJaWYgKF9fbXV0ZXhfdHJ5bG9jayhsb2NrKSkgewogCQlpZiAo dXNlX3d3X2N0eCAmJiB3d19jdHgpCi0JCQlfX3d3X211dGV4X3dha2V1cF9mb3JfYmFja29mZihs b2NrLCB3d19jdHgpOworCQkJX193d19tdXRleF9jaGVja193YWl0ZXJzKGxvY2ssIHd3X2N0eCk7 CiAKIAkJZ290byBza2lwX3dhaXQ7CiAJfQpAQCAtNzg5LDEwICs4MzQsMTMgQEAgX19tdXRleF9s b2NrX2NvbW1vbihzdHJ1Y3QgbXV0ZXggKmxvY2ssIGxvbmcgc3RhdGUsIHVuc2lnbmVkIGludCBz dWJjbGFzcywKIAkJd2FpdGVyLnd3X2N0eCA9IE1VVEVYX1BPSVNPTl9XV19DVFg7CiAjZW5kaWYK IAl9IGVsc2UgewotCQkvKiBBZGQgaW4gc3RhbXAgb3JkZXIsIHdha2luZyB1cCB3YWl0ZXJzIHRo YXQgbXVzdCBiYWNrIG9mZi4gKi8KKwkJLyoKKwkJICogQWRkIGluIHN0YW1wIG9yZGVyLCB3YWtp bmcgdXAgd2FpdGVycyB0aGF0IG11c3Qga2lsbAorCQkgKiB0aGVtc2VsdmVzLgorCQkgKi8KIAkJ cmV0ID0gX193d19tdXRleF9hZGRfd2FpdGVyKCZ3YWl0ZXIsIGxvY2ssIHd3X2N0eCk7CiAJCWlm IChyZXQpCi0JCQlnb3RvIGVycl9lYXJseV9iYWNrb2ZmOworCQkJZ290byBlcnJfZWFybHlfa2ls bDsKIAogCQl3YWl0ZXIud3dfY3R4ID0gd3dfY3R4OwogCX0KQEAgLTgxNCw3ICs4NjIsNyBAQCBf X211dGV4X2xvY2tfY29tbW9uKHN0cnVjdCBtdXRleCAqbG9jaywgbG9uZyBzdGF0ZSwgdW5zaWdu ZWQgaW50IHN1YmNsYXNzLAogCQkJZ290byBhY3F1aXJlZDsKIAogCQkvKgotCQkgKiBDaGVjayBm b3Igc2lnbmFscyBhbmQgd291bmQgY29uZGl0aW9ucyB3aGlsZSBob2xkaW5nCisJCSAqIENoZWNr IGZvciBzaWduYWxzIGFuZCBraWxsIGNvbmRpdGlvbnMgd2hpbGUgaG9sZGluZwogCQkgKiB3YWl0 X2xvY2suIFRoaXMgZW5zdXJlcyB0aGUgbG9jayBjYW5jZWxsYXRpb24gaXMgb3JkZXJlZAogCQkg KiBhZ2FpbnN0IG11dGV4X3VubG9jaygpIGFuZCB3YWtlLXVwcyBkbyBub3QgZ28gbWlzc2luZy4K IAkJICovCkBAIC04MjMsOCArODcxLDggQEAgX19tdXRleF9sb2NrX2NvbW1vbihzdHJ1Y3QgbXV0 ZXggKmxvY2ssIGxvbmcgc3RhdGUsIHVuc2lnbmVkIGludCBzdWJjbGFzcywKIAkJCWdvdG8gZXJy OwogCQl9CiAKLQkJaWYgKHVzZV93d19jdHggJiYgd3dfY3R4ICYmIHd3X2N0eC0+YWNxdWlyZWQg PiAwKSB7Ci0JCQlyZXQgPSBfX3d3X211dGV4X2xvY2tfY2hlY2tfc3RhbXAobG9jaywgJndhaXRl ciwgd3dfY3R4KTsKKwkJaWYgKHVzZV93d19jdHggJiYgd3dfY3R4KSB7CisJCQlyZXQgPSBfX3d3 X211dGV4X2NoZWNrX2tpbGwobG9jaywgJndhaXRlciwgd3dfY3R4KTsKIAkJCWlmIChyZXQpCiAJ CQkJZ290byBlcnI7CiAJCX0KQEAgLTg2OSw3ICs5MTcsNyBAQCBfX211dGV4X2xvY2tfY29tbW9u KHN0cnVjdCBtdXRleCAqbG9jaywgbG9uZyBzdGF0ZSwgdW5zaWduZWQgaW50IHN1YmNsYXNzLAog CWxvY2tfYWNxdWlyZWQoJmxvY2stPmRlcF9tYXAsIGlwKTsKIAogCWlmICh1c2Vfd3dfY3R4ICYm IHd3X2N0eCkKLQkJd3dfbXV0ZXhfc2V0X2NvbnRleHRfc2xvd3BhdGgod3csIHd3X2N0eCk7CisJ CXd3X211dGV4X2xvY2tfYWNxdWlyZWQod3csIHd3X2N0eCk7CiAKIAlzcGluX3VubG9jaygmbG9j ay0+d2FpdF9sb2NrKTsKIAlwcmVlbXB0X2VuYWJsZSgpOwpAQCAtODc4LDcgKzkyNiw3IEBAIF9f bXV0ZXhfbG9ja19jb21tb24oc3RydWN0IG11dGV4ICpsb2NrLCBsb25nIHN0YXRlLCB1bnNpZ25l ZCBpbnQgc3ViY2xhc3MsCiBlcnI6CiAJX19zZXRfY3VycmVudF9zdGF0ZShUQVNLX1JVTk5JTkcp OwogCW11dGV4X3JlbW92ZV93YWl0ZXIobG9jaywgJndhaXRlciwgY3VycmVudCk7Ci1lcnJfZWFy bHlfYmFja29mZjoKK2Vycl9lYXJseV9raWxsOgogCXNwaW5fdW5sb2NrKCZsb2NrLT53YWl0X2xv Y2spOwogCWRlYnVnX211dGV4X2ZyZWVfd2FpdGVyKCZ3YWl0ZXIpOwogCW11dGV4X3JlbGVhc2Uo JmxvY2stPmRlcF9tYXAsIDEsIGlwKTsKLS0gCjIuMTQuMwoKX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX18KZHJpLWRldmVsIG1haWxpbmcgbGlzdApkcmktZGV2 ZWxAbGlzdHMuZnJlZWRlc2t0b3Aub3JnCmh0dHBzOi8vbGlzdHMuZnJlZWRlc2t0b3Aub3JnL21h aWxtYW4vbGlzdGluZm8vZHJpLWRldmVsCg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-path: Received: from mail-co1nam03on0057.outbound.protection.outlook.com ([104.47.40.57]:50284 "EHLO NAM03-CO1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1756600AbeFSI0A (ORCPT ); Tue, 19 Jun 2018 04:26:00 -0400 From: Thomas Hellstrom To: dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org, peterz@infradead.org Cc: linux-graphics-maintainer@vmware.com, pv-drivers@vmware.com, Ingo Molnar , Jonathan Corbet , Gustavo Padovan , Maarten Lankhorst , Sean Paul , David Airlie , Davidlohr Bueso , "Paul E. McKenney" , Josh Triplett , Thomas Gleixner , Kate Stewart , Philippe Ombredanne , Greg Kroah-Hartman , linux-doc@vger.kernel.org, linux-media@vger.kernel.org, linaro-mm-sig@lists.linaro.org, Thomas Hellstrom Subject: [PATCH 1/3] locking: WW mutex cleanup Date: Tue, 19 Jun 2018 10:24:43 +0200 Message-Id: <20180619082445.11062-2-thellstrom@vmware.com> In-Reply-To: <20180619082445.11062-1-thellstrom@vmware.com> References: <20180619082445.11062-1-thellstrom@vmware.com> MIME-Version: 1.0 Content-Type: text/plain Sender: linux-media-owner@vger.kernel.org List-ID: From: Peter Ziljstra Make the WW mutex code more readable by adding comments, splitting up functions and pointing out that we're actually using the Wait-Die algorithm. Cc: Ingo Molnar Cc: Jonathan Corbet Cc: Gustavo Padovan Cc: Maarten Lankhorst Cc: Sean Paul Cc: David Airlie Cc: Davidlohr Bueso Cc: "Paul E. McKenney" Cc: Josh Triplett Cc: Thomas Gleixner Cc: Kate Stewart Cc: Philippe Ombredanne Cc: Greg Kroah-Hartman Cc: linux-doc@vger.kernel.org Cc: linux-media@vger.kernel.org Cc: linaro-mm-sig@lists.linaro.org Co-authored-by: Thomas Hellstrom Signed-off-by: Thomas Hellstrom --- Documentation/locking/ww-mutex-design.txt | 12 +- include/linux/ww_mutex.h | 28 ++--- kernel/locking/mutex.c | 202 ++++++++++++++++++------------ 3 files changed, 145 insertions(+), 97 deletions(-) diff --git a/Documentation/locking/ww-mutex-design.txt b/Documentation/locking/ww-mutex-design.txt index 34c3a1b50b9a..2fd7f2a2af21 100644 --- a/Documentation/locking/ww-mutex-design.txt +++ b/Documentation/locking/ww-mutex-design.txt @@ -32,10 +32,10 @@ the oldest task) wins, and the one with the higher reservation id (i.e. the younger task) unlocks all of the buffers that it has already locked, and then tries again. -In the RDBMS literature this deadlock handling approach is called wait/wound: +In the RDBMS literature this deadlock handling approach is called wait/die: The older tasks waits until it can acquire the contended lock. The younger tasks needs to back off and drop all the locks it is currently holding, i.e. the -younger task is wounded. +younger task dies. Concepts -------- @@ -56,9 +56,9 @@ Furthermore there are three different class of w/w lock acquire functions: * Normal lock acquisition with a context, using ww_mutex_lock. -* Slowpath lock acquisition on the contending lock, used by the wounded task - after having dropped all already acquired locks. These functions have the - _slow postfix. +* Slowpath lock acquisition on the contending lock, used by the task that just + killed its transaction after having dropped all already acquired locks. + These functions have the _slow postfix. From a simple semantics point-of-view the _slow functions are not strictly required, since simply calling the normal ww_mutex_lock functions on the @@ -220,7 +220,7 @@ mutexes are a natural fit for such a case for two reasons: Note that this approach differs in two important ways from the above methods: - Since the list of objects is dynamically constructed (and might very well be - different when retrying due to hitting the -EDEADLK wound condition) there's + different when retrying due to hitting the -EDEADLK die condition) there's no need to keep any object on a persistent list when it's not locked. We can therefore move the list_head into the object itself. - On the other hand the dynamic object list construction also means that the -EALREADY return diff --git a/include/linux/ww_mutex.h b/include/linux/ww_mutex.h index 39fda195bf78..f82fce2229c8 100644 --- a/include/linux/ww_mutex.h +++ b/include/linux/ww_mutex.h @@ -6,7 +6,7 @@ * * Copyright (C) 2004, 2005, 2006 Red Hat, Inc., Ingo Molnar * - * Wound/wait implementation: + * Wait/Die implementation: * Copyright (C) 2013 Canonical Ltd. * * This file contains the main data structure and API definitions. @@ -28,9 +28,9 @@ struct ww_class { struct ww_acquire_ctx { struct task_struct *task; unsigned long stamp; - unsigned acquired; + unsigned int acquired; #ifdef CONFIG_DEBUG_MUTEXES - unsigned done_acquire; + unsigned int done_acquire; struct ww_class *ww_class; struct ww_mutex *contending_lock; #endif @@ -38,8 +38,8 @@ struct ww_acquire_ctx { struct lockdep_map dep_map; #endif #ifdef CONFIG_DEBUG_WW_MUTEX_SLOWPATH - unsigned deadlock_inject_interval; - unsigned deadlock_inject_countdown; + unsigned int deadlock_inject_interval; + unsigned int deadlock_inject_countdown; #endif }; @@ -102,7 +102,7 @@ static inline void ww_mutex_init(struct ww_mutex *lock, * * Context-based w/w mutex acquiring can be done in any order whatsoever within * a given lock class. Deadlocks will be detected and handled with the - * wait/wound logic. + * wait/die logic. * * Mixing of context-based w/w mutex acquiring and single w/w mutex locking can * result in undetected deadlocks and is so forbidden. Mixing different contexts @@ -195,13 +195,13 @@ static inline void ww_acquire_fini(struct ww_acquire_ctx *ctx) * Lock the w/w mutex exclusively for this task. * * Deadlocks within a given w/w class of locks are detected and handled with the - * wait/wound algorithm. If the lock isn't immediately avaiable this function + * wait/die algorithm. If the lock isn't immediately available this function * will either sleep until it is (wait case). Or it selects the current context - * for backing off by returning -EDEADLK (wound case). Trying to acquire the + * for backing off by returning -EDEADLK (die case). Trying to acquire the * same lock with the same context twice is also detected and signalled by * returning -EALREADY. Returns 0 if the mutex was successfully acquired. * - * In the wound case the caller must release all currently held w/w mutexes for + * In the die case the caller must release all currently held w/w mutexes for * the given context and then wait for this contending lock to be available by * calling ww_mutex_lock_slow. Alternatively callers can opt to not acquire this * lock and proceed with trying to acquire further w/w mutexes (e.g. when @@ -226,14 +226,14 @@ extern int /* __must_check */ ww_mutex_lock(struct ww_mutex *lock, struct ww_acq * Lock the w/w mutex exclusively for this task. * * Deadlocks within a given w/w class of locks are detected and handled with the - * wait/wound algorithm. If the lock isn't immediately avaiable this function + * wait/die algorithm. If the lock isn't immediately available this function * will either sleep until it is (wait case). Or it selects the current context - * for backing off by returning -EDEADLK (wound case). Trying to acquire the + * for backing off by returning -EDEADLK (die case). Trying to acquire the * same lock with the same context twice is also detected and signalled by * returning -EALREADY. Returns 0 if the mutex was successfully acquired. If a * signal arrives while waiting for the lock then this function returns -EINTR. * - * In the wound case the caller must release all currently held w/w mutexes for + * In the die case the caller must release all currently held w/w mutexes for * the given context and then wait for this contending lock to be available by * calling ww_mutex_lock_slow_interruptible. Alternatively callers can opt to * not acquire this lock and proceed with trying to acquire further w/w mutexes @@ -256,7 +256,7 @@ extern int __must_check ww_mutex_lock_interruptible(struct ww_mutex *lock, * @lock: the mutex to be acquired * @ctx: w/w acquire context * - * Acquires a w/w mutex with the given context after a wound case. This function + * Acquires a w/w mutex with the given context after a die case. This function * will sleep until the lock becomes available. * * The caller must have released all w/w mutexes already acquired with the @@ -290,7 +290,7 @@ ww_mutex_lock_slow(struct ww_mutex *lock, struct ww_acquire_ctx *ctx) * @lock: the mutex to be acquired * @ctx: w/w acquire context * - * Acquires a w/w mutex with the given context after a wound case. This function + * Acquires a w/w mutex with the given context after a die case. This function * will sleep until the lock becomes available and returns 0 when the lock has * been acquired. If a signal arrives while waiting for the lock then this * function returns -EINTR. diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c index 2048359f33d2..412b4fc08235 100644 --- a/kernel/locking/mutex.c +++ b/kernel/locking/mutex.c @@ -243,6 +243,17 @@ void __sched mutex_lock(struct mutex *lock) EXPORT_SYMBOL(mutex_lock); #endif +/* + * Wait-Die: + * The newer transactions are killed when: + * It (the new transaction) makes a request for a lock being held + * by an older transaction. + */ + +/* + * Associate the ww_mutex @ww with the context @ww_ctx under which we acquired + * it. + */ static __always_inline void ww_mutex_lock_acquired(struct ww_mutex *ww, struct ww_acquire_ctx *ww_ctx) { @@ -281,26 +292,53 @@ ww_mutex_lock_acquired(struct ww_mutex *ww, struct ww_acquire_ctx *ww_ctx) DEBUG_LOCKS_WARN_ON(ww_ctx->ww_class != ww->ww_class); #endif ww_ctx->acquired++; + ww->ctx = ww_ctx; } +/* + * Determine if context @a is 'after' context @b. IOW, @a is a younger + * transaction than @b and depending on algorithm either needs to wait for + * @b or die. + */ static inline bool __sched __ww_ctx_stamp_after(struct ww_acquire_ctx *a, struct ww_acquire_ctx *b) { - return a->stamp - b->stamp <= LONG_MAX && - (a->stamp != b->stamp || a > b); + + return (signed long)(a->stamp - b->stamp) > 0; +} + +/* + * Wait-Die; wake a younger waiter context (when locks held) such that it can + * die. + * + * Among waiters with context, only the first one can have other locks acquired + * already (ctx->acquired > 0), because __ww_mutex_add_waiter() and + * __ww_mutex_check_kill() wake any but the earliest context. + */ +static bool __sched +__ww_mutex_die(struct mutex *lock, struct mutex_waiter *waiter, + struct ww_acquire_ctx *ww_ctx) +{ + if (waiter->ww_ctx->acquired > 0 && + __ww_ctx_stamp_after(waiter->ww_ctx, ww_ctx)) { + debug_mutex_wake_waiter(lock, waiter); + wake_up_process(waiter->task); + } + + return true; } /* - * Wake up any waiters that may have to back off when the lock is held by the - * given context. + * We just acquired @lock under @ww_ctx, if there are later contexts waiting + * behind us on the wait-list, check if they need to die. * - * Due to the invariants on the wait list, this can only affect the first - * waiter with a context. + * See __ww_mutex_add_waiter() for the list-order construction; basically the + * list is ordered by stamp, smallest (oldest) first. * * The current task must not be on the wait list. */ static void __sched -__ww_mutex_wakeup_for_backoff(struct mutex *lock, struct ww_acquire_ctx *ww_ctx) +__ww_mutex_check_waiters(struct mutex *lock, struct ww_acquire_ctx *ww_ctx) { struct mutex_waiter *cur; @@ -310,30 +348,23 @@ __ww_mutex_wakeup_for_backoff(struct mutex *lock, struct ww_acquire_ctx *ww_ctx) if (!cur->ww_ctx) continue; - if (cur->ww_ctx->acquired > 0 && - __ww_ctx_stamp_after(cur->ww_ctx, ww_ctx)) { - debug_mutex_wake_waiter(lock, cur); - wake_up_process(cur->task); - } - - break; + if (__ww_mutex_die(lock, cur, ww_ctx)) + break; } } /* - * After acquiring lock with fastpath or when we lost out in contested - * slowpath, set ctx and wake up any waiters so they can recheck. + * After acquiring lock with fastpath, where we do not hold wait_lock, set ctx + * and wake up any waiters so they can recheck. */ static __always_inline void ww_mutex_set_context_fastpath(struct ww_mutex *lock, struct ww_acquire_ctx *ctx) { ww_mutex_lock_acquired(lock, ctx); - lock->ctx = ctx; - /* * The lock->ctx update should be visible on all cores before - * the atomic read is done, otherwise contended waiters might be + * the WAITERS check is done, otherwise contended waiters might be * missed. The contended waiters will either see ww_ctx == NULL * and keep spinning, or it will acquire wait_lock, add itself * to waiter list and sleep. @@ -347,29 +378,14 @@ ww_mutex_set_context_fastpath(struct ww_mutex *lock, struct ww_acquire_ctx *ctx) return; /* - * Uh oh, we raced in fastpath, wake up everyone in this case, - * so they can see the new lock->ctx. + * Uh oh, we raced in fastpath, check if any of the waiters need to + * die. */ spin_lock(&lock->base.wait_lock); - __ww_mutex_wakeup_for_backoff(&lock->base, ctx); + __ww_mutex_check_waiters(&lock->base, ctx); spin_unlock(&lock->base.wait_lock); } -/* - * After acquiring lock in the slowpath set ctx. - * - * Unlike for the fast path, the caller ensures that waiters are woken up where - * necessary. - * - * Callers must hold the mutex wait_lock. - */ -static __always_inline void -ww_mutex_set_context_slowpath(struct ww_mutex *lock, struct ww_acquire_ctx *ctx) -{ - ww_mutex_lock_acquired(lock, ctx); - lock->ctx = ctx; -} - #ifdef CONFIG_MUTEX_SPIN_ON_OWNER static inline @@ -645,37 +661,73 @@ void __sched ww_mutex_unlock(struct ww_mutex *lock) } EXPORT_SYMBOL(ww_mutex_unlock); + +static __always_inline int __sched +__ww_mutex_kill(struct mutex *lock, struct ww_acquire_ctx *ww_ctx) +{ + if (ww_ctx->acquired > 0) { +#ifdef CONFIG_DEBUG_MUTEXES + struct ww_mutex *ww; + + ww = container_of(lock, struct ww_mutex, base); + DEBUG_LOCKS_WARN_ON(ww_ctx->contending_lock); + ww_ctx->contending_lock = ww; +#endif + return -EDEADLK; + } + + return 0; +} + + +/* + * Check whether we need to kill the transaction for the current lock acquire. + * + * Wait-Die: If we're trying to acquire a lock already held by an older + * context, kill ourselves. + * + * Since __ww_mutex_add_waiter() orders the wait-list on stamp, we only have to + * look at waiters before us in the wait-list. + */ static inline int __sched -__ww_mutex_lock_check_stamp(struct mutex *lock, struct mutex_waiter *waiter, - struct ww_acquire_ctx *ctx) +__ww_mutex_check_kill(struct mutex *lock, struct mutex_waiter *waiter, + struct ww_acquire_ctx *ctx) { struct ww_mutex *ww = container_of(lock, struct ww_mutex, base); struct ww_acquire_ctx *hold_ctx = READ_ONCE(ww->ctx); struct mutex_waiter *cur; + if (ctx->acquired == 0) + return 0; + if (hold_ctx && __ww_ctx_stamp_after(ctx, hold_ctx)) - goto deadlock; + return __ww_mutex_kill(lock, ctx); /* * If there is a waiter in front of us that has a context, then its - * stamp is earlier than ours and we must back off. + * stamp is earlier than ours and we must kill ourself. */ cur = waiter; list_for_each_entry_continue_reverse(cur, &lock->wait_list, list) { - if (cur->ww_ctx) - goto deadlock; + if (!cur->ww_ctx) + continue; + + return __ww_mutex_kill(lock, ctx); } return 0; - -deadlock: -#ifdef CONFIG_DEBUG_MUTEXES - DEBUG_LOCKS_WARN_ON(ctx->contending_lock); - ctx->contending_lock = ww; -#endif - return -EDEADLK; } +/* + * Add @waiter to the wait-list, keep the wait-list ordered by stamp, smallest + * first. Such that older contexts are preferred to acquire the lock over + * younger contexts. + * + * Waiters without context are interspersed in FIFO order. + * + * Furthermore, for Wait-Die kill ourself immediately when possible (there are + * older contexts already waiting) to avoid unnecessary waiting. + */ static inline int __sched __ww_mutex_add_waiter(struct mutex_waiter *waiter, struct mutex *lock, @@ -692,7 +744,7 @@ __ww_mutex_add_waiter(struct mutex_waiter *waiter, /* * Add the waiter before the first waiter with a higher stamp. * Waiters without a context are skipped to avoid starving - * them. + * them. Wait-Die waiters may die here. */ pos = &lock->wait_list; list_for_each_entry_reverse(cur, &lock->wait_list, list) { @@ -700,34 +752,27 @@ __ww_mutex_add_waiter(struct mutex_waiter *waiter, continue; if (__ww_ctx_stamp_after(ww_ctx, cur->ww_ctx)) { - /* Back off immediately if necessary. */ - if (ww_ctx->acquired > 0) { -#ifdef CONFIG_DEBUG_MUTEXES - struct ww_mutex *ww; + /* + * Wait-Die: if we find an older context waiting, there + * is no point in queueing behind it, as we'd have to + * die the moment it would acquire the lock. + */ + int ret = __ww_mutex_kill(lock, ww_ctx); - ww = container_of(lock, struct ww_mutex, base); - DEBUG_LOCKS_WARN_ON(ww_ctx->contending_lock); - ww_ctx->contending_lock = ww; -#endif - return -EDEADLK; - } + if (ret) + return ret; break; } pos = &cur->list; - /* - * Wake up the waiter so that it gets a chance to back - * off. - */ - if (cur->ww_ctx->acquired > 0) { - debug_mutex_wake_waiter(lock, cur); - wake_up_process(cur->task); - } + /* Wait-Die: ensure younger waiters die. */ + __ww_mutex_die(lock, cur, ww_ctx); } list_add_tail(&waiter->list, pos); + return 0; } @@ -771,7 +816,7 @@ __mutex_lock_common(struct mutex *lock, long state, unsigned int subclass, */ if (__mutex_trylock(lock)) { if (use_ww_ctx && ww_ctx) - __ww_mutex_wakeup_for_backoff(lock, ww_ctx); + __ww_mutex_check_waiters(lock, ww_ctx); goto skip_wait; } @@ -789,10 +834,13 @@ __mutex_lock_common(struct mutex *lock, long state, unsigned int subclass, waiter.ww_ctx = MUTEX_POISON_WW_CTX; #endif } else { - /* Add in stamp order, waking up waiters that must back off. */ + /* + * Add in stamp order, waking up waiters that must kill + * themselves. + */ ret = __ww_mutex_add_waiter(&waiter, lock, ww_ctx); if (ret) - goto err_early_backoff; + goto err_early_kill; waiter.ww_ctx = ww_ctx; } @@ -814,7 +862,7 @@ __mutex_lock_common(struct mutex *lock, long state, unsigned int subclass, goto acquired; /* - * Check for signals and wound conditions while holding + * Check for signals and kill conditions while holding * wait_lock. This ensures the lock cancellation is ordered * against mutex_unlock() and wake-ups do not go missing. */ @@ -823,8 +871,8 @@ __mutex_lock_common(struct mutex *lock, long state, unsigned int subclass, goto err; } - if (use_ww_ctx && ww_ctx && ww_ctx->acquired > 0) { - ret = __ww_mutex_lock_check_stamp(lock, &waiter, ww_ctx); + if (use_ww_ctx && ww_ctx) { + ret = __ww_mutex_check_kill(lock, &waiter, ww_ctx); if (ret) goto err; } @@ -869,7 +917,7 @@ __mutex_lock_common(struct mutex *lock, long state, unsigned int subclass, lock_acquired(&lock->dep_map, ip); if (use_ww_ctx && ww_ctx) - ww_mutex_set_context_slowpath(ww, ww_ctx); + ww_mutex_lock_acquired(ww, ww_ctx); spin_unlock(&lock->wait_lock); preempt_enable(); @@ -878,7 +926,7 @@ __mutex_lock_common(struct mutex *lock, long state, unsigned int subclass, err: __set_current_state(TASK_RUNNING); mutex_remove_waiter(lock, &waiter, current); -err_early_backoff: +err_early_kill: spin_unlock(&lock->wait_lock); debug_mutex_free_waiter(&waiter); mutex_release(&lock->dep_map, 1, ip); -- 2.14.3