From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754873AbdEHOOP (ORCPT ); Mon, 8 May 2017 10:14:15 -0400 Received: from mx0a-00082601.pphosted.com ([67.231.145.42]:33553 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754179AbdEHOOM (ORCPT ); Mon, 8 May 2017 10:14:12 -0400 Authentication-Results: vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=none action=none header.from=fb.com; Subject: Re: Large latency on blk_queue_enter To: =?UTF-8?Q?Javier_Gonz=c3=a1lez?= , Ming Lei References: <1656B440-3ECA-4F2B-B95C-418CF0F347E9@lightnvm.io> <20170508122738.GC5696@ming.t460p> <76E35BA3-FEC9-46D6-B36F-554F464FA9ED@lightnvm.io> CC: Christoph Hellwig , Dan Williams , , Linux Kernel Mailing List From: Jens Axboe Message-ID: Date: Mon, 8 May 2017 08:13:56 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <76E35BA3-FEC9-46D6-B36F-554F464FA9ED@lightnvm.io> Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: 8bit X-Originating-IP: [216.160.245.98] X-ClientProxiedBy: CY4PR0101CA0016.prod.exchangelabs.com (10.171.218.29) To CY4PR15MB1192.namprd15.prod.outlook.com (10.172.177.14) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 3520263c-05de-44c4-dd6c-08d4961c7a67 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(201703131423075)(201703031133081);SRVR:CY4PR15MB1192; X-Microsoft-Exchange-Diagnostics: 1;CY4PR15MB1192;3:9AVK0YwQyJIpX1Qp8BWUpeYmZoKxe6AyM6dEhes32Y62jo3nL9lGuMpuJuV8tf/ifcntnbw0NSBzmf8y8rRqW92PxXtVMlkXF851PyIz+pOHNK5pqL6G1XAGawyRFU1EFJxX4pgLftadu+/FDcO7dQztls0q+fscFGlBx9YvicLq5VJLwRzUWTCY2W7xeu6WgcOUGlMCRW7UV9MvZhz2CiS36E7/cAKiRqB6N5eD2U2nITcS/2yVHhkrBJMJfrP8FnhPeknDdRWOBu4+OtfWuJUTFwmPP9cXU/16x9zIQLMGXLpnRUeVjyZmdfdu+I+WqjaluCWbi+QYNBi2od4zUQ==;25:E0OlICWno6ilhE24ux7d53scP75htMQyQb8UeB2sGV0/74WPXHGUqBs4jaMzUpX0Ye+z9TXngSFrJl2Wx7CEvAJg3PlqYIB72/StILdD10Q6Irot6EZrOFNZk0LrLn6lDmD2mrliEduAppVCXhmhOM0eaX+/+SofuDo9qGfSvDwu+7RSjIh6WQhWu8BsQvSc+ZKTQ9i023iVahA+u2mTnxxsIA4vstXJZm8tH2AlGPTsLQyfo29xypKH3MLTPZkdClyBFgqqFvXYKMekFQWEU8+8U1jXJOLiCTjYgrAdxfFFmbfQ6t0wn6iatSLtQpqBB6K1C0DLfGNsXgnRWyCTOkKUc7bqn7CWN10hX4FN/hjSgSRxUtuLLEbFqTJ4XOAOr4ZVBsZOVN1wWKwh6OhrJQ9c48Dh5NIVY3V5U8DPRn1aB2awRjhLKiKlAvrQ8owrrSreQ3MxQLC+oYPwHY0QhDlAPcaEuiBL44gCbhFd2kA= X-Microsoft-Exchange-Diagnostics: 1;CY4PR15MB1192;31:xAcAbeHGok6I8bb+OHvrwoSnGq6L4a0V04RKKS4jHiOuTMzqrcl/rzrtnPcMfCANEuDydkHYMdU9FsHpJwuPBVEkXZ/tM4TFIjucB17TAS9GJ2CvziSRLPuK/2oftTaQgaUceAsNK+UnoM36EGb7BSSYMWc6DOvgAYt4C8xrA1Az6BHJplV8nghOgJEh+pSMm7X1w3UvBJmqlIdJ6ds13NLS8Jeik8GFRyTuMPxeA4g9Oe47RQCIfuDLlHBfEMqiyD4eD5Rwaquo7KX9FJq8VQ==;20:Q1xZNXnFHTWaCoOsyGa1jb1O37WUyNgVBrB8LZUsTeMsgUvzyFk/wqn0vwttajaNAvlo6h+vcId8UsA1X/D2/TUJJSu2f16Oh/Q5al+mGrSEKFVbzumpvLtT2dEhnptA/atloLXWtW6R+a07BwMpKZDPAU8wimweX5W3QBWLS3r3Yc3J1/ukOe75nabwx4SZe/V5KSLb1LtPnZ+WPpp4db4Pxq/Z8sFCggsmM7HabPaWsZlvM1Dj7MvZ87wuHe2kTvEn9rJJG5iy5RrY476PVPnO7bkdQ5DG5yWgJgn8ZAEYLAYSgagnBLq4tWuS5i/UhpD4X3F6j0Es8pBg6K0noPP4dB+04dTtgzyuot/LWucG8WmI7zFfhpxxbKobKVCqPufCUdG0hUySVUfVy3KUJBJlDz5rFxHibfGTQ77vhOGM0Z/HOjKUdwrm7I4aoXqcg6pgY3Y0I1abkSlFxNIJOu4OWsMu2qHrr26cWbuUf0fbJiNqGT//sPc6EcfJCZE6 X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040450)(601004)(2401047)(5005006)(8121501046)(93006095)(93001095)(3002001)(10201501046)(6041248)(20161123558100)(20161123562025)(20161123555025)(20161123560025)(20161123564025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(6072148);SRVR:CY4PR15MB1192;BCL:0;PCL:0;RULEID:;SRVR:CY4PR15MB1192; X-Microsoft-Exchange-Diagnostics: 1;CY4PR15MB1192;4:/mcSJAfKgxBUZOzi/IwkJvNbVstX8x0spgYUIDDmkUeuGnBB4cUCSCP7e5oCCYKMONs5U+fsZ87sN1VbKdJK9WgUXHXZw2LMihnNB3kSyEmG6qcHUqiJ48Z19s96xNDzZkb/fRvNyph5BOUWEiDI8WfFjSkLTZlkHRxA2dNiwjKP6hkfORKSa2NEgM/4AxN7URTjvU5hlVpVaKkpunl2uTxtMcIqECTdbwpce8fHkYAaYTlsQILbEVT9x5DPk4TaN7i2vWIWcTtF7i0EnqFciIEXQVHMNzGZebu+qRFKQd30QM00m/YDCj0oDMA1p97qIoFE+5N5wSwJZDXQZgQ+k4bNKpIy8tmj+LIpopM1GEp1AQYhvRpmhRhJzIwHF/1LZrOQRZCB6czRUJnsuc+z4ojansRocvajijpmqTLFfIsx6kgyYo1kLchXRWJXwC49cwIhVqySh97sQEW/D0vq8y+QvZjClMUo829HtsB+6aMksuLmYllsOV9ZYUIJoeYrimEJTRUQiHuyrD3wMdeL1waRHBMU5/d6XEdVVUNng90VAAamUQj3h7btrLL0w7TlD/81fzATUotBO6xwmJWoqcJMnK1JGc+MKR4ajwUDTmdDlqUbEUsaCodQjL3vbPTxSdYfNy0nQR10HjcGRL1kXnEN5j6R10C8B4xkrD1PUx/1kjI9PRU1oQSaYBqBkNJ/KJuOklD2/Com5bPl16GXPihHw7Cv8STXRFX97RR2a1IJbKEIUBZMOr5wJsVXXJWSnPFAsN8ExxhJXuGk3RT6slEq+mMoyB35nVGp0evzM1w= X-Forefront-PRVS: 0301360BF5 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6009001)(6049001)(39450400003)(39840400002)(39850400002)(39400400002)(39410400002)(24454002)(377454003)(7736002)(6666003)(53936002)(305945005)(4326008)(8676002)(3846002)(5660300001)(90366009)(53546009)(6486002)(77096006)(6246003)(86362001)(2950100002)(36756003)(31696002)(33646002)(83506001)(229853002)(38730400002)(189998001)(25786009)(65826007)(2870700001)(6116002)(47776003)(66066001)(65956001)(54356999)(478600001)(117156002)(42186005)(50986999)(76176999)(65806001)(64126003)(81166006)(50466002)(4001350100001)(2906002);DIR:OUT;SFP:1102;SCL:1;SRVR:CY4PR15MB1192;H:[192.168.1.154];FPR:;SPF:None;MLV:sfv;LANG:en; X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1;CY4PR15MB1192;23:mfbmvIvOO28eznWhuPAS3wHS++FbEgR/USlRL?= =?Windows-1252?Q?CE6IIl7tUqLCfWPseOWQ/FtRUbNoPeA91e8VLDgBTuqlxxXfu/fEbmbS?= =?Windows-1252?Q?+s0q+W6ktAze5ZaE2+GibXeBEq1g4VedUJ6EupLnh3xmkeh6P152UAQp?= =?Windows-1252?Q?pm0nhzd/cg9HqW+7TVBcOfpOSQPXCRbWe09S1xK2WL/NY9y2HV9D7JO1?= =?Windows-1252?Q?Jf+fXIFNBlCJwRID9h0k78WtclEJ2ZBikUbdxB1MinI/GxRMbnyzZLfu?= =?Windows-1252?Q?GXjtUDf+7qkkD3rvkPFeD52GF5wejM2C7BHe6quzjfRls/ySXTondXhs?= =?Windows-1252?Q?5LzlzoIUlnuJud3/r4sCV1c7IeUQ2JYX6jm402fcwIZ4Y0rDII5Xw7/e?= =?Windows-1252?Q?dv25xSEAku1JKAwrYilqsbirV1N67Oc418AZtu3Us8Lf6YM/i4NP29qc?= =?Windows-1252?Q?7x1Be+sXjQQHOT7S4wDRHrqyEPqLbDqkDdRuY4TUij2MilaD+GslZ79V?= =?Windows-1252?Q?zCjNcZ+Eqb+gi0XqBl97MnsDXVceiEuaIyg9RIYrHf7oECu1xiZxa0QE?= =?Windows-1252?Q?GBFRhK70gQfZHc+lUdXUtS+NtODkH3RL3vWh+N7MiGGUkYLOjVRn2ccH?= =?Windows-1252?Q?lLZTeNP9FA2QWr9BvBmtNpNzBCPDoKBMmjpvJFzXdjTepfobdaUyyCDy?= =?Windows-1252?Q?DeqTRynQJGXkYENTpQbvD5dctmjw2klX8zMjiqf5NxMQ4TLUH4CXz/og?= =?Windows-1252?Q?VT9w7gJanHd9mCH+X1QSzTjJWGt4nksOfOvhYnijqhyAqOtkcMJe8qhP?= =?Windows-1252?Q?rmEDn+Jq33FG2sRhwceKtN4cNhSDI2tfRXB23rOjIIBM7Id29nZ6eNpc?= =?Windows-1252?Q?UF2+4VJPJ3a31BL0UPpi/Mie/TkSUqFJQbpxA37iHdKVdPd9qrTdzl11?= =?Windows-1252?Q?/OPWDJie34yB9dRw62EkcRMHUFr5MSaGryzYotZT5+F+nGV+XtMPx1Hy?= =?Windows-1252?Q?kguhR5x2l+1vyn7XdA8Fc/qhQlKUXwSp+9yACsGyLGN2fde/NJvkEW1A?= =?Windows-1252?Q?cI7waM3qqr9QtsajwBG+VBDCalugzbN7Px4fTVBqCuH+FlLgz3stzrVs?= =?Windows-1252?Q?cMyuOLp2Tqtv0hTBMguUSFlYmOzZZ0QqwlxzrSOQR5GkmVOnZX9nI7ku?= =?Windows-1252?Q?e8CJXX1kW+ig/950w6K2OvBbmFHcwpQy8J/pRuv1XhZefJ9+FpJ2ZC0G?= =?Windows-1252?Q?VmpP+BD8y7CblZORRKZ+f2wsLiHr0m/rZeqnuS/lPIVObUmSvHh5N7t7?= =?Windows-1252?Q?l9L610gUCu9klrYgItYtWPvmuIMJmhYBEll3naU2ZzyPuCfOVkc6K1W8?= =?Windows-1252?Q?KeWHbqHQHIE?= X-Microsoft-Exchange-Diagnostics: 1;CY4PR15MB1192;6:GnG8aqLGJsVEqzANJXsP4XPwP+XfzOoJRy/Z+q3PluxWMEUQroeCyQmgSjR/6O1ouxLknOUCRYivpg5B/UweetGmTqosfVgRYZEovf7+UMrocWxa7Qem+KQkRxYN8KieaHuL6/CDUut2ivbHIQu3x7213jdK6+pls/NRDdEGnI4AWo2XvGj3oM+YUqbohreZK28LAZOKH6AC6oKHjikho0HULFhNKT9v3X62SXOkY8VZ5AnuLwQW510t98nPd3DXd0RN+oYX0FyhbB6uD/mtVwHHH3v+gaAPZK/n8BvCWKJAGk+a6YkwjnxbOx7TMaOTdQRVtpVBH4eHbDS1kltpyrPpEqzOlDnVkwuf2Sw0/X/PSRGhZP3M6CDGD3Hvvtv605DE15orN/rVF1i4N+kZQ39imr4KhCAJOXzQasEqr5BR4nHeG3nlAhN9Qijvah6cFS1velzMxrNiX8IyrN/g/+RexnNmBNlg/uYe3PZCrJEHl3QXh6QWxLdDH0fW9kj7gKLWjPefF+icyZNLwu6dGA==;5:EKVDs4MH9riJ5L7fN/LWyGC/xRUJXjBzdmusWZBsGaxGcdK3XIuXyc1E92HIl9Gizb/3V1lEDAl3+vxCXXjh3VWCcdapr4kfdxzstwCMV0fRUpo+uanX/aDWZdM+ePd0Bu87w8442Dlm5PMqkQMLdB+uE5s/ese5NpeAVMKOD/8=;24:/ekuqGf8hJ/e6urb22tGnoVHSbC+xSj/CLLIEcLdWTmadYA65x3khBKT9uulUYijQKL2MM0WVg8vOvjCrXUgwqo2RnORTrtH7fuGR1E5gzQ= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;CY4PR15MB1192;7:HrdCpNLKoaYNR+3/QXzggpTOQcXUXy8zopr344BqpCXSG3ujSEf4gJM/ErRe0pS0bNc6P52cJxrTW4OqNDQ6du05x1GIMK4Rxp34U0HRe7nTxg9rHgA7xiY6wWVlJ3n7l1SR/Ic5/zFT0G7aWkNVu6rZjd4PfzFfwTWZZFmcjIqatmaVCOsVbNtXr5jnKRVAHGtB/jaZ4otl0C33qfKf8Yb3dWBICExoiTPVGOMHP/MvGjymBrf7ocwzwRPeQLxODAtf9MTAyfAMvnvM5KjDFu6OVC1vKK+nSBosiaYVNBRXVnkQCT2fWMJTiskS001aDL/GTMTTUag+IJe/KILA3A==;20:pHC8UWeWbw73WViYARKdBdJg2E7Kl5E6Yve/LwRtXJs1aUn8X/MmbeiHV9liu4ffY5FQlcpLvaKRyZlvJliKNmkg3uXIXFCh/3/drJ+RYTwXW9VB6o8ftD3GJW2/NaFPx6GOde8U+zyF9QWk38DAl8HCT46oliVRLgQG/eTqBCc= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 May 2017 14:14:02.0778 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR15MB1192 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-05-08_10:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 05/08/2017 07:44 AM, Javier González wrote: >> On 8 May 2017, at 14.27, Ming Lei wrote: >> >> On Mon, May 08, 2017 at 01:54:58PM +0200, Javier González wrote: >>> Hi, >>> >>> I find an unusual added latency(~20-30ms) on blk_queue_enter when >>> allocating a request directly from the NVMe driver through >>> nvme_alloc_request. I could use some help confirming that this is a bug >>> and not an expected side effect due to something else. >>> >>> I can reproduce this latency consistently on LightNVM when mixing I/O >>> from pblk and I/O sent through an ioctl using liblightnvm, but I don't >>> see anything on the LightNVM side that could impact the request >>> allocation. >>> >>> When I have a 100% read workload sent from pblk, the max. latency is >>> constant throughout several runs at ~80us (which is normal for the media >>> we are using at bs=4k, qd=1). All pblk I/Os reach the nvme_nvm_submit_io >>> function on lightnvm.c., which uses nvme_alloc_request. When we send a >>> command from user space through an ioctl, then the max latency goes up >>> to ~20-30ms. This happens independently from the actual command >>> (IN/OUT). I tracked down the added latency down to the call >>> percpu_ref_tryget_live in blk_queue_enter. Seems that the queue >>> reference counter is not released as it should through blk_queue_exit in >>> blk_mq_alloc_request. For reference, all ioctl I/Os reach the >>> nvme_nvm_submit_user_cmd on lightnvm.c >>> >>> Do you have any idea about why this might happen? I can dig more into >>> it, but first I wanted to make sure that I am not missing any obvious >>> assumption, which would explain the reference counter to be held for a >>> longer time. >> >> You need to check if the .q_usage_counter is working at atomic mode. >> This counter is initialized as atomic mode, and finally switchs to >> percpu mode via percpu_ref_switch_to_percpu() in blk_register_queue(). > > Thanks for commenting Ming. > > The .q_usage_counter is not working on atomic mode. The queue is > initialized normally through blk_register_queue() and the counter is > switched to percpu mode, as you mentioned. As I understand it, this is > how it should be, right? That is how it should be, yes. You're not running with any heavy debugging options, like lockdep or anything like that? -- Jens Axboe