From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751902AbcHKXWM (ORCPT ); Thu, 11 Aug 2016 19:22:12 -0400 Received: from mail-sn1nam01on0091.outbound.protection.outlook.com ([104.47.32.91]:60096 "EHLO NAM01-SN1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751022AbcHKXWK (ORCPT ); Thu, 11 Aug 2016 19:22:10 -0400 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=waiman.long@hpe.com; Message-ID: <57AD0898.7030506@hpe.com> Date: Thu, 11 Aug 2016 19:22:00 -0400 From: Waiman Long User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.12) Gecko/20130109 Thunderbird/10.0.12 MIME-Version: 1.0 To: Dave Hansen CC: Thomas Gleixner , Ingo Molnar , "H. Peter Anvin" , , , Jiang Liu , Borislav Petkov , Andy Lutomirski , Prarit Bhargava , Scott J Norton , Douglas Hatch , Randy Wright , John Stultz Subject: Re: [RESEND PATCH v4] x86/hpet: Reduce HPET counter read contention References: <1470853770-37625-1-git-send-email-Waiman.Long@hpe.com> <57ACD2DE.6080306@intel.com> In-Reply-To: <57ACD2DE.6080306@intel.com> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [72.71.243.167] X-ClientProxiedBy: BLUPR0301CA0035.namprd03.prod.outlook.com (10.162.113.173) To AT5PR84MB0307.NAMPRD84.PROD.OUTLOOK.COM (10.162.138.29) X-MS-Office365-Filtering-Correlation-Id: 662f9ab7-bc04-4295-2e8d-08d3c23e501f X-Microsoft-Exchange-Diagnostics: 1;AT5PR84MB0307;2:ar4L5EN6eVe7Y+QHkgscGoF/z+ahi4V0jLlvwwbLgTbSdke7HvKNH4iZtD8JXIBnAaK0mYPAjrYJEFOuxQ2oM1qtPC2ough96PKCU7Eq5hSKiknXqCYWw9761ti8aDlbP3a0Gce3Z4gsqQueToKe/TX1y0bQ0GJfWdxaRuT1gWXkqkMZadMewRlKlx4Q51zm;3:p6gi87BXfY766xh9HwN9JuKSp0TmHV/zqc+kQnA3H3W2oEakCUEG/QYZxziRBzA6sJkbgKU1JsBYi+kmOee3skrohT4R3T76rkzM793EzMRFiZosjcG7oQwdV4SZTb/Q X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:AT5PR84MB0307; X-Microsoft-Exchange-Diagnostics: 1;AT5PR84MB0307;25:L1qCPR8U06j9YPFxApL+ACWQrmv5Pvee9xBtY4izFJ6nTtkSESZVXEv/AUL6qJmBdr1EUQ1cjXwqOLbE9L5m8L+eV+a7qxRNzEM+9+g11c7XDqragiG8FqyJeUcjBFAwLlWz2NDmhEP43ya2qeicMus7FThKvfTiDP5kbQdScL0XLSZQyVe3uAZQE0gHd7iFRBUZi/rXzZWdfogn9Djd+S8oZdXwyWFY+zAD1jE2FjphzVbEV7znWdAfViqvV9muK8l/8WVq3MoGXqJW/nfK3w66sKdnjuvzT+zsLwwfBBPR10B87/r5Iqo4a381A9jLd8RjXWWYHyXnFM+X1e48PXq96CZpzcmlnl+yiWsAYOEGaAR6b0IfP8murB6LzxSVPFod1dWyboXu171UL3zcJpbghyV5wKWKq7H6+GCIBylvCfHwmiAdCvZ8pl9CygFR5oTITJBx3QQVyx+nS6Aao7nmPwuOGcaneS1Xd8L2KtbgH1CGeGRu1gE9pkbu4LmjpK/20i6AA5H7ectR2kdGnqqnEQiEenyUu9NltzPNTUoewCfDV3XtEmFbgiaoOPY0hdMRM4f9/SzhKXGs7ZjFXpqvW+pR2kDJSlr6beNRBixDtYh5Lf8MkoNGXOFOqD4odwDZ9ZX49ZsivpVXPPee8ug0fAOcQMAP748kzzMyPcuN3e2S/lb1BA+52YF0NgqDjRRaRmyuvInYdXlBR35bzeUiTWweY5YX3weYnBmqBLc= X-Microsoft-Exchange-Diagnostics: 1;AT5PR84MB0307;31:Oz1hnUmENyVkfhmAdXlEADzcj9SbiXhbEpu1ewfR1LtY5dx894aMHRpunBZx54sq2eRlVJ+ST/5uvXm5Eog6pqGM150pqok81u9NVB4chdJIXtU0hUm8CZpvuceqXbcF7B8JH5DLFyaNl8MxSECxAkOLkwDfOTS01IAKgHs5Drj6L3Y9nF3qjV6fAT0l0kJv8TTMVzmr8us8XoDcy5w9CmUQErMri9js1+LUKEzfzgI=;20:aghSIFgMIkFeO/A8MZX5SYvhY7tTBNunSil2uyp71VcmFsMr029TBG7CysC0p8ynjGzeZuE1Jd/rWva4VVOOVKI85sSTkB8Et9yLJVYBgcGPUhUcNAvVeNRp7Dx/illkOb8/AhFHJMSJFvh+rGrXriRVIYYxczB0Ttb+8FYhQzXjFIyqGhIrPyWMH+GmzRyMwHTHvmGWCvJJ+1nbozcNqIvBdzsOjzXumoQDtGaML/J5ZbDxTUQi1ReySQu4gigP/qEFFDDxROxMoCmM86XmxyWaWW1YAAs2sCMH068R8nmfMit6j8+vl38CM1J3IaSf3YYBJTA7e+E1X2DXDVeeCXZo2BDYI90ufEH8T42Ux+EqmIX+eIDSFAF5oOncKMiIhr1vq8U6hl2gxH/cd1FOP9rKlgmxFMgjq002N9WHII9wpHW4Y4pe6qL2hUQEGau8UP6Sn0GnzI8/6ShKXoyPNQTTwulBEm8fnng00Mx7g+JnIgnShPLXvnaim828t5nz X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046)(6055026);SRVR:AT5PR84MB0307;BCL:0;PCL:0;RULEID:;SRVR:AT5PR84MB0307; X-Microsoft-Exchange-Diagnostics: 1;AT5PR84MB0307;4:XMrRgZ3+JvV8UJx+VkaAVQCj+k02iEBhJxlLlXSTGuM1OJIuOJx/4PsT94dU60pTAb09KwykC+q9cNH9z++MvFstULmma53KcxuETTDHeu/2AaB7XcGpjq4WeOg5EENDhC8wLZOL+QNMIc7p8c3UT7iOXNxpkIQCbREB5pwsYlctLBJRj6mHgyhMJvlH0mZC4/P6JQxwXkhvCS76jaXVQg/CrsngX+eCaaykPb7R/rF+M0mdDint6lzJ4gjLFgPnP3q3LBZc9tJ4rCH8DsvWKD44fOsQdlmcPGDTukvUmPy6+WRiWLmOaHrZQgqOfIZVCl0H90zQ2w0dPc94XgERntw7I/ZnINj724M6qoQ84b9tpSqhINv3eHpwgFUO+akbJNphJdTuG1wzcFHp0Fwb4g== X-Forefront-PRVS: 0031A0FFAF X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6009001)(6049001)(7916002)(377454003)(24454002)(199003)(189002)(42186005)(86362001)(101416001)(65816999)(87266999)(50986999)(76176999)(105586002)(54356999)(65806001)(65956001)(117156001)(47776003)(66066001)(33656002)(106356001)(7846002)(80316001)(2906002)(68736007)(3846002)(230700001)(81156014)(92566002)(6116002)(83506001)(189998001)(586003)(81166006)(4001350100001)(110136002)(77096005)(19580395003)(59896002)(97736004)(64126003)(2950100001)(23746002)(7416002)(50466002)(36756003)(305945005)(4326007)(7736002)(8676002)(217873001);DIR:OUT;SFP:1102;SCL:1;SRVR:AT5PR84MB0307;H:[192.168.142.185];FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1;AT5PR84MB0307;23:RtwANPY2r1/PNp1E1fbrCzfa7tUvXy/N3eA3p?= =?Windows-1252?Q?/ic9DNBdHVGMGlADUj48zrjTOJDSL1FzRi2NtRiIKCQ9uDYgvB9s8tzq?= =?Windows-1252?Q?vgEhZZOV5bWE5Dzro+aQGi9F+krrPlHhJaqJQ6SKbrOKa4D3YZT5OUAA?= =?Windows-1252?Q?WlnGYUBQTt69IsqndmwJktcViPZZc6lhEE2+O4DwcnYsuM32TcYEfCNR?= =?Windows-1252?Q?nYSSAttOzettKXunNP6dxaqJy0g0lDDrjbtSW5SR+uOaQXVJw+TubZ0p?= =?Windows-1252?Q?UsO1P6eL9vs1t3acbNoOzg0eHcXCUXc7d28DIMFb3/bDFGmUDuKlN0Jn?= =?Windows-1252?Q?cyL2z1I5zvFxNH72BWKXfLX0aE/QmjipRN1gfcDVfi4i4pmZyt59sC3H?= =?Windows-1252?Q?dS+8DSOBA0xO/NZzSIfXiHkL22TVBHirHiHo4Ijhhx+Hs7Zqdp3hNy9z?= =?Windows-1252?Q?afv5E2Z8OiHlra8Z53TPL8BFXYVVKuzSKcEgMJCT1wLOF01f/OgRn8ZX?= =?Windows-1252?Q?ukXTJ1mG0Io8HbxWGVQcdUDH8HjJIvj4s8lkVIRktBfSImchYx37BpIU?= =?Windows-1252?Q?VlxM/ryaVnZaAd1af2h1TR6eZ1OE3h9ntw+6Vriud/GAK7hRu5rShKhv?= =?Windows-1252?Q?w3kaVVEHdzA6rvpOS3dHLITZTx3JbggHNA0XWUYBPYkOFk8TI94wc64a?= =?Windows-1252?Q?brLegDQKTFaAJQ3Ws8PSvC4dUqLVAbDEefJo0eK9eHE0viVQL5LZtXLk?= =?Windows-1252?Q?Ba5ALsISLMUO8DHxebF5RDNwfl0HcSGky0mKhgMOrLTnad6ISiVpGfjp?= =?Windows-1252?Q?vwZN9b7CEiaSFf2WYd1+O+7iNVnJX1ToXLEU6rIclnzDdX9t482vn/Of?= =?Windows-1252?Q?dxBGqLqrweKbRcZtzr3X8lrq7DlovFwUJHfwJdNdNM4DOcL8UHItK5ww?= =?Windows-1252?Q?8OYMP2RZcPfEgAV0XCPjozdERbdTj6GcBODi7M0VOKuoThR4SqAWQvrW?= =?Windows-1252?Q?s1bKg5teqoAC8Yz/n4/5sZpkSeNfJkwXaAC819QmAILPfbe4/Qduq6ni?= =?Windows-1252?Q?VZB8IBnlhmu4WjrdRUCA8yQ9rgJy6ShO/UNzk4a9pz/tfw58m45S49Dc?= =?Windows-1252?Q?p5UJkbbBye4sAu6PoWbRzGj4KgktnOLxopkXme2d5TQoMfbGjYTG8qjO?= =?Windows-1252?Q?/O+mZNg+jWA7EOLzBoHFh3H1980tI98diXdvXVWSEGNOa9c+tvt4BO5V?= =?Windows-1252?Q?yNptROTgXdM90lMKd21n8SxrzqHDRBX+N+hQzoXBvs51QUmkk7guC0//?= =?Windows-1252?Q?bwvbw5ehGo0O7/lNgQ+xs9yNtB7IQn8p3UJ78N7feaJWs16QH1pHVvW1?= =?Windows-1252?Q?AOy+UZpdSsAClID4HQiI4k5T5lnsGXb5DVUJHHhEVCt1aaQGnY9A9g?= =?Windows-1252?Q?=3D?= X-Microsoft-Exchange-Diagnostics: 1;AT5PR84MB0307;6:0g2uvAPyb7jZZKvU2Fh/0tAJ0f4cH9UB9khZwDPXYQCZ3911bZCwpFPlQtpqHNzSmU6tyAInsZbn/r7QQjGDCSSQB2itAOYw3LxGOqCyzrtUNVMBYcjggGyzp4WxNC2ncqssoG7iRr+KbE7s1DStCnRai1YLPcUAazVv2uNpfrVPZDwh0yXoKKAUYrqi8sT9FrpBVp0NdvQk1lkRsejDLM43Vzzznr3j9Tgyfrvh47VaFOFBwA0WOMjvczArzoSSC+rWQFPTQZxldFF/19c6X/fRRV2+KwN+ZhH5GAxRac9OiJNDMYmM5Ck3yw92JYyZVsViLWPh6cAMkW/5F8dDXg==;5:bJcph2oq5l3mTPzHBqOjTOEDEX6Kw6sfk78BnXGxJSVXEfD83Vm3rX8g6EcRpdRDxk3ETyKc08G1n6dabr+wg+TUJOTF0ZYu+tz4Y6IHzX+iqcJAoZcMNI79jJ4AhNJ9dijpsz0DY2jgew7tQuoLvA==;24:zVrU4j2qCmQL3oR53vgXYWZGb5NRANjAXnVlVpCpTjN6VU0uRS7Gdq+hhYjQDAFSU+V/oMyCO6Jc4VZp9b8vuvbtVrT7u/DbHrVRYdcrivw=;7:YBBOXLzBS8iA0W562j2gtwgbULNEtd07sgq6l4+qWMT+VqUavg/B6EnjxNwZuX6HOJYV4Tx3cwnOcrkjwVtjwBtYQNTYohDA10w1mq8tlHhMz0QcHo2gf7ItwbMvni1g6/KJNGmXDHn60eSf5ZocQqL4CUD+WbexW36A1tXd+IkjYt04upXOu+1c+BF3u6bCliL9lnm7a/nfgbGboOLsUGeqi6AhBUa+3mdmzoKhPjPE+DmoqHLLTyCkvhBuwTnc SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: hpe.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Aug 2016 23:22:06.8605 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: AT5PR84MB0307 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 08/11/2016 03:32 PM, Dave Hansen wrote: > On 08/10/2016 11:29 AM, Waiman Long wrote: >> +static cycle_t read_hpet(struct clocksource *cs) >> +{ >> + int seq; >> + >> + seq = READ_ONCE(hpet_save.seq); >> + if (!HPET_SEQ_LOCKED(seq)) { > ... >> + } >> + >> + /* >> + * Wait until the locked sequence number changes which indicates >> + * that the saved HPET value is up-to-date. >> + */ >> + while (READ_ONCE(hpet_save.seq) == seq) { >> + /* >> + * Since reading the HPET is much slower than a single >> + * cpu_relax() instruction, we use two here in an attempt >> + * to reduce the amount of cacheline contention in the >> + * hpet_save.seq cacheline. >> + */ >> + cpu_relax(); >> + cpu_relax(); >> + } >> + >> + return (cycle_t)READ_ONCE(hpet_save.hpet); >> +} > It's a real bummer that this all has to be open-coded. I have to wonder > if there were any alternatives that you tried that were simpler. What do you mean by "open-coded"? Do you mean the function can be inlined? > Is READ_ONCE()/smp_store_release() really strong enough here? It > guarantees ordering, but you need ordering *and* a guarantee that your > write is visible to the reader. Don't you need actual barriers for > that? Otherwise, you might be seeing a stale HPET value, and the spin > loop that you did waiting for it to be up-to-date was worthless. The > seqlock code, uses barriers, btw. The cmpxchg() and smp_store_release() act as the lock/unlock sequence with the proper barriers. Another important point is that the hpet value is visible to the other readers before the sequence number. This is what the smp_store_release() is providing. cmpxchg is an actual barrier, even though smp_store_release() is not. However, the x86 architecture will guarantee the writes are in order, I think. > Also, since you're fundamentally reading a second-hand HPET value, does > that have any impact on the precision of the HPET as a timesource? Or, > is it so coarse already that this isn't an issue? There can always be unexpected latency in the returned time value, such as an interrupt or NMI. I think as long as the time won't go backward, it should be fine. Cheers, Longman