From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753352AbcFOTLA (ORCPT ); Wed, 15 Jun 2016 15:11:00 -0400 Received: from mail-bn1bbn0102.outbound.protection.outlook.com ([157.56.111.102]:35340 "EHLO na01-bn1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751061AbcFOTK6 (ORCPT ); Wed, 15 Jun 2016 15:10:58 -0400 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=waiman.long@hpe.com; Message-ID: <5761A48F.1040304@hpe.com> Date: Wed, 15 Jun 2016 14:55:11 -0400 From: Waiman Long User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.12) Gecko/20130109 Thunderbird/10.0.12 MIME-Version: 1.0 To: Dave Chinner CC: Peter Zijlstra , Ingo Molnar , , , , , , , , Davidlohr Bueso , Jason Low , Scott J Norton , Douglas Hatch Subject: Re: [RFC PATCH-tip 6/6] xfs: Enable reader optimistic spinning for DAX inodes References: <1465927959-39719-1-git-send-email-Waiman.Long@hpe.com> <1465927959-39719-7-git-send-email-Waiman.Long@hpe.com> <20160614230613.GB26977@dastard> In-Reply-To: <20160614230613.GB26977@dastard> Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [71.168.64.62] X-ClientProxiedBy: CY1PR13CA0017.namprd13.prod.outlook.com (10.162.30.155) To CS1PR84MB0312.NAMPRD84.PROD.OUTLOOK.COM (10.162.190.30) X-MS-Office365-Filtering-Correlation-Id: 4170719c-4d97-4954-1dd1-08d3954e9e1d X-Microsoft-Exchange-Diagnostics: 1;CS1PR84MB0312;2:7dhYYKCIvnXQJtANH/VfmrmevCPrIqjzVG2l4wXtNOunkOhiSfJe7upikgvP88OqI0A/AUie/aTmZq2tZgrH2AMcmMoka6Az8WfeA6TC+XJWp6YCF73qZf+1lj1uAwl2oEfbOxO5I88OT8OiWMRIS1Qm9UtmQi3fDcEVCtJGJetlOL7SLQjNBwuXT0ShhWhc;3:n7j3liX9p9FxNKKCzFRPd8gtTf0bFo0IBp2LcqnGVZaY4dN9hbrmjbC+FC6c4A4JAVPM6kWG2nVSSbTgeW35UmytjCQSuKA1aHttyPCJqHqY55d3kLss9LHeSLVYIvYd X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:CS1PR84MB0312; X-Microsoft-Exchange-Diagnostics: 1;CS1PR84MB0312;25:TFMTD87Z8HTSfzzNNsZZ/hb8ITblGUvuQxIG7Uu9TTJL3ZD1m5Vg63HEhnyqikse1ULu0iiZ/YTKvqQ3XEiyK+drBmbo4jj0Iy+iaeywrNbE4BT9z+yspUq7FnodlDVeWv9TJUjk5xmrWIAybUL/XSrpxv9/CmEqSG5u2YcPm5HqjJ7Q9evlDmm3pFBFebEy/HfHq9/+JETkK9rTWCThP2N/KOLU8we69ADxnFF598tmvjx/w2uZk8o9SMERPsMSy1GclPSxJGHesRgB+158yHXGQ6/UCklS7g8u6gTq9K260QkfN2GPN0soCMhoguZ5ENVH5PiyZ6j1wb1AWMybGZRxF1Z/rf7ffbHoLWJADu7IponKI83noBCtA1uMUt21n7nxAfayBI4E3+SERwTUxsYLyLhWD+9s36uj7Rp3Bh76zu4qVxkE+QJWLJaCfuxwNhAmmL8Et22uqff4e4nt4oj1ofR2NXNVtIwYSxO8140e5FLc5CMIOnxzLm3dSeH7mtitVWYFWBPpeVb2ru5E+py8+XDMgJql71pwGCBkK9ybF2SGeiMPwX3H4816pdAB133xc88J/lRdNZdgsj8DDpeoWT4y0j7JndfdR5zoUI/U1JvqjZZIflPwCV40ZwXLPsP7FPT/9yXFDQztAwCZRMS1fObIQ3YWy1pA0++nHfqhQLWXXC+BMT+SYmhj81+J1/HhCWi9tF6dNy+3wCsrtFUtubfIAprmSFoirfytgtZa/XWupm5sitIg8MwNnqr2 X-Microsoft-Exchange-Diagnostics: 1;CS1PR84MB0312;20:QTU0mk5Fn+SzjtUYBu29Cs0jKoeh+eXmtD8jJK30C6FJEXVsbFzHGsQsCmU1g6iskhzZV0SVmIZR4/5+Ct/A0Pb9RPJTmPCRA0d9eN7FukJH/7JmQ5D5kIW73B/CzVEs0jqfTIyD/JGO7Nrf/rF3R/vBG/RY4tYj3SmIH+8BMuz22AQDkIDJVi6YlZuX5Tc8A6cUKmSwBCbjkL6rwXEL4vLqECNjCkPASA5HIK325T8+dUppDsy7QouDOMXImoZaIsR+VOMuAp1u6/FrWrezf4j85PsCjv4JXm33dOPZ95k2kznIDcVq9pDB9p4xOVJqOmX81G3538Fcw9EXQwIwvafTdrNZVYr0DT0ZxkuESqOSZNhUqC5B/wBqwcYlCVTw/nwb+laFVZJGtprC191S32bQzSwdAbl2hUASIMPDOqXfdUBYnKIoSTtaoniitJpQC91B+mfa5kXIJpaEHexNUI5wDx+bu5EL66XteGEZQ/wivj6gge7decI1kD3QrxW1 X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(227479698468861)(17755550239193); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(5005006)(8121501046)(3002001)(10201501046);SRVR:CS1PR84MB0312;BCL:0;PCL:0;RULEID:;SRVR:CS1PR84MB0312; X-Microsoft-Exchange-Diagnostics: 1;CS1PR84MB0312;4:XSzoqwZvRtYioPxq4+DCKyLrC+Ua4o4bnosGaq8ccdgTICFpWZQahurl/yMUFH5GNzPTNuAyzV3QEM9YlrRnAyGmlmX4y/UO6waRZOOM/+XaS2Qn73KSwYI87pQNg8RO1BbxZ0X2dyoAxjPEIG0uR8+qODwbTTTUHH30EV9yVo+7659/bY1v/oFguYfjSjKFdcphlHy4odT35/QbWD4mfSwGTf+yKYHuiFk4pbxToYmRHDXBWVW9OQ3fwYHavwh5Bwg65oXwG0rnyAXbRLkmPtV+cnAkg5mnyToBIs25IR6EgvZl0UjcbX0wF9yDfttmBrRhcPGrp7q09Vfu0so9VDexHFE8g+buUvhgHtiSvbnn6++3O8WmO9ZNM8ZNfH+df5iseiRLyWaIIbWdGWC6JcCw00VG8sGLwDGys860jSS7RQ6ddUV5CEhbGTvFMiS4P1K3M5EWylX9PMRnjThTvA== X-Forefront-PRVS: 09749A275C X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6049001)(6009001)(7916002)(24454002)(377454003)(199003)(189002)(106356001)(59896002)(189998001)(101416001)(5008740100001)(92566002)(5004730100002)(86362001)(19580395003)(117156001)(65956001)(19580405001)(80316001)(2950100001)(47776003)(77096005)(42186005)(33656002)(83506001)(23756003)(105586002)(50466002)(64126003)(65816999)(87266999)(65806001)(2906002)(36756003)(4326007)(230700001)(3846002)(586003)(110136002)(81156014)(81166006)(66066001)(6116002)(8676002)(76176999)(8666005)(68736007)(4001350100001)(50986999)(54356999)(97736004)(7059030);DIR:OUT;SFP:1102;SCL:1;SRVR:CS1PR84MB0312;H:[192.168.142.154];FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?iso-8859-1?Q?1;CS1PR84MB0312;23:htHS4A9Og3HZxly1nezkl8L4/WJ26kcDI0G4TxQ?= =?iso-8859-1?Q?cU9Kx8sI3omy4MczEFkolpyruGJryndSpN2wAmJmzSJ7oRSIfxHB70UBMS?= =?iso-8859-1?Q?evYVtDhv6MaO6apBsbg66vw1VxSBD5Xz2hOS8y82HaitCSK9dD54bbYixx?= =?iso-8859-1?Q?TSTH4AqdfKtzuSt4crDY+SJi+wPZc9m4Y79rWhFHVwVv52YtwPEfryq9U2?= =?iso-8859-1?Q?vr9dgdCBI3QbYz2OmtYji+/eSzu+heXEOkjAxWGUMVuhy7Yif8Th2PuW0b?= =?iso-8859-1?Q?QkBjvd2nkY8OvwmIz6OiEYkL+Dgn95N7rDAUHo2Pt6v/BFworYCsDAUbFE?= =?iso-8859-1?Q?FqsUiJf+4kHj5bjmS+7PGoJ/1mqklnlZn6vg2u+839xNvXMH3LRZn0qqTi?= =?iso-8859-1?Q?aYIw6zTeGOrHaUFQsPy7WomUmvon+q0ThpVuJYBGw7CvNi4hWAx3yordG2?= =?iso-8859-1?Q?zXLBtB63UG3Gm/eryKlqQiosf15VLIHso5cXI0xU0I2JzGbr9eVVfJPx6C?= =?iso-8859-1?Q?qgAfMZxDRDSCUOfycs3MgmQD3GXTVNqe74qeheNeZkVz9PvKwiK4hDu8mp?= =?iso-8859-1?Q?/RWBGBBNfH8WVWlqbm91YYtP4xITl8t+SOrQb7vYm1O0qPd3pU+oF5ldrn?= =?iso-8859-1?Q?O3WMVxKKebNvTPWOmlP0m/Kz5isoTPNKnQBlh4JFsv1VOPrGiifNpLwWIx?= =?iso-8859-1?Q?R1uoV69jdFb1VnOz2tB3lfNxYISLa9NS7KKPuly60LhPGxqJevtlhDdsY+?= =?iso-8859-1?Q?W9DSXS7xSMFskqcR+uzo2UcK3/5ojMseMd38l/lCB1TzebWgh1y+L7vtFq?= =?iso-8859-1?Q?2mCyuCPt2cQFd91M2D0YWurNeBlZlTLf7zWPXjpOEfBNEz29fDHi/C9U9q?= =?iso-8859-1?Q?/GwRlOHwGy7E3HWriTco32x4WPYRNvMxHcWcanLB/951dTYUvSupObuOeE?= =?iso-8859-1?Q?SbwBIm2q3DjNsMmyDXID2FmzgclYwNrlI0mbzX4T701ADTqJkMbQj6xdhi?= =?iso-8859-1?Q?y2SAo9vVLprxGrMgNLX+0Ln9slD+9L17eKkj112W//RDpDi/d79UDyYBoO?= =?iso-8859-1?Q?S4uFD91WAr1CCKmEoyQBu+Ut4NpOmjqTgrmL2WwMT1/F9zu1mDqzjjutE1?= =?iso-8859-1?Q?mfFSJqe7BQvJln7mGqbfMQD1GM4ALYFN8r4V60Oi/uVMfbf14XGMCH0QEC?= =?iso-8859-1?Q?NmlZDnZGfDCmCdN1RY3VCNljaVynkWYynpn3UIvJOIhCmoEyZdRmXIySse?= =?iso-8859-1?Q?OB57V7zXDVDoEBy0J0fcHp7cmBENUQN08LU/5kedEYl92lHU5U8roOPsuO?= =?iso-8859-1?Q?NAPBrpC1yP0ZsTyNl+pv5KECBSWMCYl34jrnubWEHblMWoUds6UMalmR5x?= =?iso-8859-1?Q?Vc+nco4/XHiXDFu/mn3alzz/mDPF9+aNcJPwdjlGqT8jkGtG9Fg=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1;CS1PR84MB0312;5:WpM1b+6OStF7ytnKQ3blIAz3TsmBGBq8eoM6X7Quo5M/eWe4FV3CHpuSqRESdSbtzM6xWCQZHn4odZsqRxjXINclwYeCrBh//5I06f9BlXzMbs8VNep0hLwYd07Ony8BVkXAAI6K7K1/3Dy3fGJ2pQ==;24:Av3fMJzQRzdHJYMg3O20QpdEzPW1joCBqXCSx3Gct7jbuDSHFzW2lX2LLxNMu2KYH/zFk9hQa57KfugWOR13OxTn1j84AvaAHGtRWo7+quM=;7:Ev6WVPmnQrAly1j6wa7hGsdJ/4bcbaDDOaVe4OCF5syOqBOUwfdVZafbTlZ8jo9ryK25kY2jnBZA+szMEFCnvKq+Sd5lFj3EDACyx+fnlLeB+O2EoOShYQZl3rGKD/lXfmzydf6ni+sF6yutoe9tX16K1c5e56rGxLyxygaUTn/VyD/pMEGxskkPZkOAE7nS7+d4m1/b7+xeDRUYQNJ0w21P4eIoLDA6N98yEgROEFE= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: hpe.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jun 2016 18:55:21.1909 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: CS1PR84MB0312 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 06/14/2016 07:06 PM, Dave Chinner wrote: > On Tue, Jun 14, 2016 at 02:12:39PM -0400, Waiman Long wrote: >> This patch enables reader optimistic spinning for inodes that are >> under a DAX-based mount point. >> >> On a 4-socket Haswell machine running on a 4.7-rc1 tip-based kernel, >> the fio test with multithreaded randrw and randwrite tests on the >> same file on a XFS partition on top of a NVDIMM with DAX were run, >> the aggregated bandwidths before and after the patch were as follows: >> >> Test BW before patch BW after patch % change >> ---- --------------- -------------- -------- >> randrw 1352 MB/s 2164 MB/s +60% >> randwrite 1710 MB/s 2550 MB/s +49% >> >> Signed-off-by: Waiman Long >> --- >> fs/xfs/xfs_icache.c | 9 +++++++++ >> 1 files changed, 9 insertions(+), 0 deletions(-) >> >> diff --git a/fs/xfs/xfs_icache.c b/fs/xfs/xfs_icache.c >> index 99ee6ee..09f284f 100644 >> --- a/fs/xfs/xfs_icache.c >> +++ b/fs/xfs/xfs_icache.c >> @@ -71,6 +71,15 @@ xfs_inode_alloc( >> >> mrlock_init(&ip->i_iolock, MRLOCK_BARRIER, "xfsio", ip->i_ino); >> >> + /* >> + * Enable reader spinning for DAX nount point >> + */ >> + if (mp->m_flags& XFS_MOUNT_DAX) { >> + rwsem_set_rspin_threshold(&ip->i_iolock.mr_lock); >> + rwsem_set_rspin_threshold(&ip->i_mmaplock.mr_lock); >> + rwsem_set_rspin_threshold(&ip->i_lock.mr_lock); >> + } > That's wrong. DAX is a per-inode flag, not a mount wide flag. This > needs to be done once the inode has been fully initialised and > IS_DAX(inode) can be run. > > Also, the benchmark doesn't show that all these locks are being > tested by this benchmark. e.g. the i_mmaplock isn't involved in > the benchmark's IO paths at all. It's only taken in page faults and > truncate paths.... > > I'd also like to see how much of the gain comes from the iolock vs > the ilock, as the ilock is nested inside the iolock and so > contention is much rarer.... This patch has now been superseded by a second one where changes to the xfs code is no longer needed. The new patch will enable reader spinning for all rwsem and dynamically disable it depending on past history. > As it is, I'm *extremely* paranoid when it comes to changes to core > locking like this. Performance is secondary to correctness, and we > need much more than just a few benchmarks to verify there aren't > locking bugs being introduced.... The core rwsem locking logic hasn't been changed. There are some minor changes, however, on what RWSEM_WAITING_BIAS value to use that requires more eyeballs to make sure that it hasn't introduced any new bug. Cheers, Longman