From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9DD08C282CD for ; Mon, 3 Mar 2025 16:12:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:In-Reply-To:Content-Type: MIME-Version:References:Message-ID:Subject:Cc:To:From:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=oBr3Q35PYOX8KLcyP5TMvc9JdpscxzNbBR95ep2biiE=; b=0rebvLy63ykY4F7brePWASGIuN YscnpvAoYOQg5NZqhURgV5Ak33ZD3hKqqlSYGe/xReam8URL8at3Jqz+PoYZqmWd1B1sUd+Qnhf3S 4GvXGfjskJKlVPOV95fVmBqUcomIFxakgQuaNBBQPXNh2T2dMdForVdEWYr7TStcGalzgufMTg0de ze2+AZB7bD0S2H/ZWim7reEll7J50TJPiLCJe5YbwqnjBE/ksSXleDdYX1vmnV3MqVqCBC5OMAf9/ QP540QiuTivDv+YQQ2AM1Oj8+2eimMV4IqmOL0A/yBykPqo/M0GN1VfT/tJH6M/nVlNOdcoN8n9pV JS+Mj9GA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tp8PX-00000001SRx-1NC5; Mon, 03 Mar 2025 16:12:55 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tp82E-00000001Nj3-3i9P for linux-nvme@bombadil.infradead.org; Mon, 03 Mar 2025 15:48:50 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=oBr3Q35PYOX8KLcyP5TMvc9JdpscxzNbBR95ep2biiE=; b=KF4dtobHd1vxFyKHtfGmYLTnJu 3geMPg49K4GbOBm2pLeLDxmdEyM8wxBYxMf0A6ZCUckskxQC8SObjg7eTENqRnD+xezhPXhoVYHt7 BIa6qhmEz9fV+4iAUt8AtSB0mCb9bo/tn0/azePHbgZeoTb9n1epLw+QhwDJCzecaA/PCHoFKXcTX YzLAysAqb1fhfLjGJtAwfR99COhkmANTeVDCN0GV88kJod+UULBOqZjVdywdyIZq8Xtd7uUnlRtjW bHDeAxxeCqrp+xFeD2K166PKbV/B9nTwf9tEdMJHfR7IP0C7C7IqpVZ1OCAMaIrfuxbxcT6EvnVve YABkLQIw==; Received: from willy by casper.infradead.org with local (Exim 4.98 #2 (Red Hat Linux)) id 1tp82C-0000000BqnG-2N01; Mon, 03 Mar 2025 15:48:48 +0000 Date: Mon, 3 Mar 2025 15:48:48 +0000 From: Matthew Wilcox To: Hannes Reinecke Cc: Vlastimil Babka , Sagi Grimberg , "linux-nvme@lists.infradead.org" , "linux-block@vger.kernel.org" , linux-mm@kvack.org Subject: Re: Kernel oops with 6.14 when enabling TLS Message-ID: References: <08c29e4b-2f71-4b6d-8046-27e407214d8c@suse.com> <509dd4d3-85e9-40b2-a967-8c937909a1bf@suse.com> <15be2446-f096-45b9-aaf3-b371a694049d@suse.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <15be2446-f096-45b9-aaf3-b371a694049d@suse.com> X-BeenThere: linux-nvme@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Linux-nvme" Errors-To: linux-nvme-bounces+linux-nvme=archiver.kernel.org@lists.infradead.org On Mon, Mar 03, 2025 at 04:39:47PM +0100, Hannes Reinecke wrote: > On 3/3/25 15:42, Matthew Wilcox wrote: > > On Mon, Mar 03, 2025 at 02:27:06PM +0000, Matthew Wilcox wrote: > > > We have a _lot_ of page types available. We should mark large kmallocs > > > as such. I'll send a patch to do that. > > > > Can you try this? It should fix the crash, at least. Not sure why the > > frozen patch triggered it. > > Still crashes: It warns, but doesn't crash! This is an improvement. > [ 63.658068] WARNING: CPU: 6 PID: 5216 at mm/slub.c:4720 > free_large_kmalloc+0x89/0xa0 > [ 63.667728] RIP: 0010:free_large_kmalloc+0x89/0xa0 > [ 63.842773] Call Trace: > [ 63.934398] kfree+0x2a5/0x340 > [ 63.987632] nvmf_connect_admin_queue+0x105/0x1a0 [nvme_fabrics > 18bfa9223bf0bd1ec571f5f45774adcc919a867e] > [ 63.987641] nvme_tcp_start_queue+0x192/0x310 [nvme_tcp > a0629454ac5200d03b72a09e4d2b1e27dfa113e9] > [ 63.987649] nvme_tcp_setup_ctrl+0xf8/0x700 [nvme_tcp > a0629454ac5200d03b72a09e4d2b1e27dfa113e9] > [ 64.043323] nvme_tcp_create_ctrl+0x2e3/0x4d0 [nvme_tcp > a0629454ac5200d03b72a09e4d2b1e27dfa113e9] > [ 64.043332] nvmf_dev_write+0x323/0x3d0 [nvme_fabrics > 18bfa9223bf0bd1ec571f5f45774adcc919a867e] > [ 64.043344] vfs_write+0xd9/0x430 > [ 64.108458] ---[ end trace 0000000000000000 ]--- > [ 64.108461] page: refcount:0 mapcount:0 mapping:0000000000000000 > index:0x2 pfn:0x5e3a > [ 64.108465] flags: 0xfffffc0000000(node=0|zone=1|lastcpupid=0x1fffff) > [ 64.108469] raw: 000fffffc0000000 0000000000000000 fffb0b48c0178e90 > 0000000000000000 > [ 64.108472] raw: 0000000000000002 0000000000000000 00000000ffffffff > 0000000000000000 > [ 64.108473] page dumped because: Not a kmalloc allocation Right. So you called kfree() on something that isn't currently kmalloced memory. Either it used to be kmalloced memory and we freed the slab that it used to be in, or it's a wild pointer. Whichever it is, that's a bug in the caller, not in slab. Why it bisected to that commit, I can't say. Maybe it changed the timing, or maybe it was just luck (whether the allocation which is now being freed is the last allocation in the slab or not). > [ 66.084156] page: refcount:0 mapcount:0 mapping:0000000000000000 > index:0x0 pfn:0x5de5 > [ 66.093770] flags: 0xfffffc0000000(node=0|zone=1|lastcpupid=0x1fffff) > [ 66.101810] raw: 000fffffc0000000 0000000000000000 dead000000000122 > 0000000000000000 > [ 66.111311] raw: 0000000000000000 0000000000000000 00000000ffffffff > 0000000000000000 > [ 66.111314] page dumped because: Not a kmalloc allocation > [ 66.112001] page: refcount:0 mapcount:0 mapping:0000000000000000 > index:0xdc pfn:0x5de3 > [ 66.137452] flags: 0xfffffc0000000(node=0|zone=1|lastcpupid=0x1fffff) > [ 66.137460] raw: 000fffffc0000000 ff45d9a24d93f420 ff45d9a24d93f420 > 0000000000000000 > [ 66.137464] raw: 00000000000000dc 0000000000000000 00000000ffffffff > 0000000000000000 It happened again ;-) > [ 66.137466] page dumped because: Not a kmalloc allocation > [ 66.138095] page: refcount:0 mapcount:0 mapping:0000000000000000 > index:0x0 pfn:0x5de5 > [ 66.180944] flags: 0xfffffc0000000(node=0|zone=1|lastcpupid=0x1fffff) > [ 66.180950] raw: 000fffffc0000000 ff45d9a24da3f420 ff45d9a24da3f420 > 0000000000000000 > [ 66.180953] raw: 0000000000000000 0000000000000000 00000000ffffffff > 0000000000000000 > [ 66.180954] page dumped because: Not a kmalloc allocation And again ... > [ 66.181672] BUG: unable to handle page fault for address: > ff40e4ea8fa50250 Oh, now it crashed. But we have so much evidence of a bug in the caller at this point that I don't think we can blame slab for falling over. If you're double-freeing something that's _not_ in a freed slab, this is the kind of thing we might expect? You need to turn on the debugging options Vlastimil mentioned and try to figure out what nvme is doing wrong.