From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756850AbcE1Dlt (ORCPT ); Fri, 27 May 2016 23:41:49 -0400 Received: from mail-bl2on0129.outbound.protection.outlook.com ([65.55.169.129]:35840 "EHLO na01-bl2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1756811AbcE1Dll (ORCPT ); Fri, 27 May 2016 23:41:41 -0400 Authentication-Results: linux.vnet.ibm.com; dkim=none (message not signed) header.d=none;linux.vnet.ibm.com; dmarc=none action=none header.from=hpe.com; Message-ID: <57491368.3030003@hpe.com> Date: Fri, 27 May 2016 23:41:28 -0400 From: Waiman Long User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.12) Gecko/20130109 Thunderbird/10.0.12 MIME-Version: 1.0 To: xinhui CC: , , Subject: Re: [PATCH] pv-qspinlock: Try to re-hash the lock after spurious_wakeup References: <1464156575-4732-1-git-send-email-xinhui.pan@linux.vnet.ibm.com> <57474114.1030506@hpe.com> <201605271033.u4RATRXc042700@mx0a-001b2d01.pphosted.com> In-Reply-To: <201605271033.u4RATRXc042700@mx0a-001b2d01.pphosted.com> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-Originating-IP: [72.71.243.249] X-ClientProxiedBy: CO2PR11CA0010.namprd11.prod.outlook.com (10.141.242.148) To CS1PR84MB0312.NAMPRD84.PROD.OUTLOOK.COM (10.162.190.30) X-MS-Office365-Filtering-Correlation-Id: a7a1af00-d9e3-4641-eea7-08d386a9f96d X-Microsoft-Exchange-Diagnostics: 1;CS1PR84MB0312;2:WO6x9uQ9IFTtkL7HHTnmsQ5yI0IGC9guY4vCR6qVh4g9pZSYytIiLxYgR4RMIv0jpgluzvGX/YIsHImIyquREgSsGRRBgXGrm9cW9e2skLrDV6hBg8S3rfx3b0AbT/YV1IW0axIocTfMPCWREFprrwK/AczEBezjmu5eLWUJ5dJi1/8KUsB1beYCn0R8Lyab;3:xmmmJaqn8jD1pW48X5l7KkE9gjE3YSRgYqlg6c7yCOaPOeSAh0ffFnhNTQNaSghbwfdVOxnPBC9gVkf2Tmxbyv23vakqbEnVsmm7E+5gZ818usFcRxcMODb/eZwIziAF;25:VXkWVNvAMfdNtPssSes4PA7OyptCdxNhozCizf0IiBKW2pGb2V64NICukv8mwwHZyOe96ZIQdDFTcjQTQTtTUnSyi0br9iddMrNUha/8S60G/E2cIA6svHUAPm4S5UvAjJqZAlM7p2eY2de24lsRq+5hUrloWMetM3EtU4bW7yEf2CEQIiFkEkb+laBQGF0XICIghQE/mnFbDFj6p3oHFQSNP2ln+YLQyuKqRWZuMAUKI4e3+CdJfDgkGOMJJ7ePx2hD+xma9URgUdU7BgPD+RDP1KCIpfIU23x/0FL/S9aNX6IzKWf8MG5YKVzfgGL0p3V5brUljBLaNL71O5usyBF1OAshtcCnD8Jnyvx1+fyZTbICBZlJTM2BU8ZJh2Q+IO9Ms/3uNVpk0VK40hRt67vjB/U4yriKabCoQszPAmc= X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:CS1PR84MB0312; X-Microsoft-Exchange-Diagnostics: 1;CS1PR84MB0312;20:udxbv+G1ZP8SY+2Rn7i6ZYXhTyZTJn/jLBGtx01eq/nt4JwzUB5GHB7ukreGAAG4jKHjZlrYuxkrW0XoHq/eVR2UVTRzgBkSmIwxr4ZnMlbgjNezlaXcKYCAnuWU0iZUfSlB/Jd1lo0zPvHmSyCY0/Y15dp1Ww9tmSJlnsi2QTw4WjGPsN6XIUsYAYMclqdhnv0+dkyhKkLaUCtQPPyTqOgRGQiTUCjk44iJHnpF1PHbyeVjTKeZQWp8CU4mQATiGSbldXaySr7hmt5X5PRvGKdk29lGSkDgrnKzAEK7ujnva/ZtVX8fWdBFuEVwwoAb3O5E2VzfOgWvOJoBmsPhXg==;4:3s1OZELDRgDbaKZQMMZLqzakVE9Mz/tPkf5kOuxXHa7yR94wh9A4hlUmaeAYi1K4U0KLA57cWhMnej3NuBQ+C/QiaxJe3rNJ985t/8+87H3wf4Oqk17nRyuge64WTjvsK/KpSTeSVBXYbAOKmlBnR8r5B0qTckuQ5iicUKJd5hpFy4xF2YhvANyf3Lc6I58VvX9z3+dQ977dNsDcS2gjzidlmhprwWjiyOinOCRuM3GV2102FDI3HCXAfj/mvLyPnfMg3PlWiRl9prjwZmYiXeIxd1nRceTD1uH41h4WHtTLvN5hWEBh+faL/B0mr0MK7jzjk5PlI64kAnrw+x7cSm5bqgrhhJAGJej08zo6sknL4Pv20aY6oSFhYzZ0UnmSVzkwkNxRzvvVZ+IORrQN0aGlqgz9DM2kE1t3cjO/d1U= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(104084551191319); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001);SRVR:CS1PR84MB0312;BCL:0;PCL:0;RULEID:;SRVR:CS1PR84MB0312; X-Forefront-PRVS: 09565527D6 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6049001)(6009001)(24454002)(52314003)(377454003)(83506001)(8676002)(4326007)(92566002)(19580405001)(81166006)(122286003)(19580395003)(2906002)(5004730100002)(110136002)(50986999)(6116002)(76176999)(54356999)(117636001)(42186005)(4001350100001)(65816999)(117156001)(2870700001)(189998001)(586003)(5008740100001)(3846002)(86362001)(65956001)(66066001)(65806001)(50466002)(64126003)(47776003)(36756003)(77096005)(2950100001)(23676002)(62816006);DIR:OUT;SFP:1102;SCL:1;SRVR:CS1PR84MB0312;H:[192.168.142.141];FPR:;SPF:None;MLV:sfv;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtDUzFQUjg0TUIwMzEyOzIzOlA0WUc0V3FKb3NZaXFEdE1WZkxIdlFadUY1?= =?utf-8?B?Z3o5OXJnSU1KSEE5ai80RTJydWkxRFJMOUpSQm5pc3pvQkY2R3RBSmlEbUJp?= =?utf-8?B?UDk4M01HS0NrcS9HVGVWUEZaMytndEo4UW9uaGZSZ3BoU1pYSVR0Y0JqWHlv?= =?utf-8?B?QUErR2l4aW12RS9TVFFJLzgvOWNnYWg3UkpyL1ltMGdMRVlUMGhCNXBRNnND?= =?utf-8?B?Q3dDRmpaNzJLajZtUk1FQkNVT3JuTnQ2dUR6MVdPci9UdmxyN3hqa0NodDBQ?= =?utf-8?B?eG91TkdFSkxrQ21jQWF1VXlrZTdhUTBWN3NQL3JTNVFnV3hBR1ZTM21nZnha?= =?utf-8?B?SWpiWjVnSkJUajhsUG1XYm9ZZUk3bW9pK00vd1YvV05DcUFyTmdRd0Y0cGVz?= =?utf-8?B?bVc4bGxLaWJxWUhocVdNRzlQNjhCeEJYc3BjbUJNNWpCMms2Z2h5U2JtN2d4?= =?utf-8?B?RDV4OVRKaVZscUxXVzQ2MlFZUGJlczVVTS9zK2FLSzl0SDlUVGcwblRiZ1Ns?= =?utf-8?B?d2xGeWIzNzEwTkQwUWc5VzNRb2FWajVNaXFncm5FR0NHU0FPNXp6YnFRREQ3?= =?utf-8?B?MzhsYkN6dnNMRjVEWEpnM3gxdDNXU1lsTFZpbTdkYWQvcjFNYkQrSHBIWVdh?= =?utf-8?B?ZUJjcVRXamdraDNXRG1MYXAxbnFiVm9mSUlhRjM5a2hOeGJGZEhVSlVHeC9F?= =?utf-8?B?YkIxdzlTYjV4L1lsMVRMdVhLTkJUZEt0ZStndkhQd2VPS0l1SkhGU2l1Znho?= =?utf-8?B?QUc2bmlmL2J6Uzk1ckNjUG9ZdGI1MDNGaHJWRHBlYUwvQkVlN1FDc0w3VU1y?= =?utf-8?B?UWl2SytZSlNkQUtaZEhsdElKSERLRzEzbWQxYnlsYjd1SVc2ZHdEcFhHSWIz?= =?utf-8?B?NnlKMW5UbmQwWTZpZXhEUzUwbGJDUWQ0OFpPeWhXS1N5ZUZycTg5R2lvK0l3?= =?utf-8?B?bXA1cmt0UHNQVHN3VU5CVG0ra2tDVEVrR01iai91VGtIb3RPOXI3L0tkY3pa?= =?utf-8?B?RFB2SkpnVlBoQnplK0cweG5sQjVoQ2lYNVdOSUQ1SitCTDZkYjJadGxESHcw?= =?utf-8?B?VG91VlVqSjN2di82a3VtWXZVRG1KSGxPR0JxcG9IUmJJSDRUaGtKbTk4UjB2?= =?utf-8?B?L2VNV054VlViREJ2ZXNqSWhkMUhWK2ZmZ05GVWRNb1hKRStUdGFsTDBNeC9H?= =?utf-8?B?UWM5aERQMStRdEVBTUJQaThqbnNCYTI2eSsweFNpM3JIVUNwN1hHajJjVk5v?= =?utf-8?B?Nm1RMmJnYVl6cW1OQzhyYzQ5c3cwREQxdmZxdmpPWWZOeDFxazBoNmI0Nk5Q?= =?utf-8?B?eC8wRnZTZ1FrSFdrRS85bVFENzkxT3pkSjFUdjl0U0RqdWxnbEpRRHU0VGQ3?= =?utf-8?B?L3BUWExKYXovdnZXN3JKSUxOV3JDQjMyRzBVbnNOMWY2KzRXNks5aHU1ZWoy?= =?utf-8?B?ZzIxaFVaYWR4eGZRcWZHT3lEeFg0WmUrRHNXNm1tc0cwWlB0NHYzUkljNmhW?= =?utf-8?Q?DBAHD7YVen8clYUwZlwzF4Vt7d/LDudtZmQzT9cdNcsW+k?= X-Microsoft-Exchange-Diagnostics: 1;CS1PR84MB0312;5:eQbQhjYlbcGF3VP61pZuuo5wsbprTmaRsN0y2yYfUM3XowyFQ5zZ90HnKOVe2gKdjXljqa7SqmReh52I9AT41bZ4D40jMMFA0RwbExAISxC/dbzW1Dj7D0qKpP3Sh9IP6Ro7ZqGIU159AheRz3g7Sg==;24:8ijFOAjVj2NtdqBIfWtK5U96LTa2tJXDvZeqW7HzTyAzijZOWHSznW+O8wQgoqaakNId0y3wFTX2cK9a9Mpi/D/hs59JeTLE/jfq6u3ub9Q=;7:lNkNUvBZcl4azsaOMj896nDuJqk7cK5GJ2k/Ywl40OM+N+prXT5hmVgvMWZ71Y7JhLNxok71+XObfNsbrHbWiBXS04LPnv2qlL/Jhv3SKFPxEpEjYfBYDn5qSDZ06NJ0sgxc56kwRq+f18CW8tip0dzu0z4rAsWXsf4UeKXstPv1Be5l1vpApDMaOC966acM SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: hpe.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 May 2016 03:41:37.3416 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: CS1PR84MB0312 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 05/27/2016 06:32 AM, xinhui wrote: > > On 2016年05月27日 02:31, Waiman Long wrote: >> On 05/25/2016 02:09 AM, Pan Xinhui wrote: >>> In pv_wait_head_or_lock, if there is a spurious_wakeup, and it fails to >>> get the lock as there is lock stealing, then after a short spin, we >>> need >>> hash the lock again and enter pv_wait to yield. >>> >>> Currently after a spurious_wakeup, as l->locked is not _Q_SLOW_VAL, >>> pv_wait might do nothing and return directly, that is not >>> paravirt-friendly because pv_wait_head_or_lock will just spin on the >>> lock then. >>> >>> Signed-off-by: Pan Xinhui >>> --- >>> kernel/locking/qspinlock_paravirt.h | 39 >>> +++++++++++++++++++++++++++++-------- >>> 1 file changed, 31 insertions(+), 8 deletions(-) >> >> Is this a problem you can easily reproduce on PPC? I have not >> observed this issue when testing on x86. >> > Hi, Waiman > I notice the spurious_wakeup count is very high when I do > benchmark tests and stress tests. So after a simple investigation, > I find pv_wait_head_or_lock() just keep loops. > That shouldn't happen in normal case. When testing on x86, I typically get the following stat data for an over-commited guest: pv_lock_slowpath=9256211 pv_lock_stealing=36398363 pv_spurious_wakeup=311 pv_wait_again=294 pv_wait_early=3255605 pv_wait_head=173 pv_wait_node=3256280 The queue head don't call pv_wait that often. There are a bit of spurious wakeup, but it is mostly caused by lock stealing. How long is a cpu_relax() in PPC takes? > Here is my story, in my pv-qspinlcok patchset V1&&v2, pv_wait on > ppc ignore the first two parameters of *ptr and val, that makes > lock_stealing hit too much. The pvqspinlock code does depend on pv_wait() doing a final check to see if the lock value change. The code may not work reliably without that. > and when I change SPIN_THRESHOLD to a small value, system is very much > unstable because waiter will enter pv_wait quickly and no one will > kick waiter's cpu if > we enter pv_wait twice thanks to the lock_stealing. > So what I do in my pv-qspinlcok patchset V3 is that add if (*ptr > == val) in pv_wait. However as I mentioned above, then spurious_wakeup > count is too high, that also means our cpu > slice is wasted. The SPIN_THRESHOLD should be sufficiently big. A small value will cause too many waits and wake-up's which may not be good. Anyway, more testing and tuning may be needed to make the pvqspinlock code work well with PPC. Cheers, Longman