From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from fhigh-b5-smtp.messagingengine.com (fhigh-b5-smtp.messagingengine.com [202.12.124.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4C74B267B02; Sun, 10 Aug 2025 17:57:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.156 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754848643; cv=none; b=duRNqDp0Vg12LObLWBhvtqW8UzbxklvcY0UOFBe6LjM9F9Ru7G4WCOS+EchuKnfQitRx74x15qSNEDIJC1XfJ4fpmQBC3cwljJGI8YIm5m/YdCy7xmwDyL8zmdkOnrcVQmFYVRWMOZFdacxpxxpTtd1Vo1dUEOc3+1q2RsQrtMo= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754848643; c=relaxed/simple; bh=fSO2pKtQJLGHg+uw3sLgbIV1Rv915LeTxSDp+iPlEGY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=QQmoIdb0y4B7p8qXmOkI38+xFqUpuK6WP4v2kUzyABY693/wiIiqirviGSHrv9b9LsjHXzHhelI8P4DeeRUUA8T9peCfbr0stnmB7ATS3NsWD6MHtYA9kpxaaZWqQgzbT39Wmvs+XF55RKz3Yd/YBeBIXgZzLYngRTt8wakdAmc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=bzzt.net; spf=pass smtp.mailfrom=bzzt.net; dkim=pass (2048-bit key) header.d=bzzt.net header.i=@bzzt.net header.b=Qka6HBPB; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=ibFpBTjG; arc=none smtp.client-ip=202.12.124.156 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=bzzt.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bzzt.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bzzt.net header.i=@bzzt.net header.b="Qka6HBPB"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="ibFpBTjG" Received: from phl-compute-10.internal (phl-compute-10.internal [10.202.2.50]) by mailfhigh.stl.internal (Postfix) with ESMTP id EC79F7A005D; Sun, 10 Aug 2025 13:57:18 -0400 (EDT) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-10.internal (MEProxy); Sun, 10 Aug 2025 13:57:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bzzt.net; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm1; t=1754848638; x=1754935038; bh=CKJHwhQNQy+ROAE4lhfUgG6FBAvwCYWmzuMLynqXNDE=; b= Qka6HBPBqe5fFkfsB1QsgZOj02B+CQ+76ZlodrPFL+Ch8r3VIJk5Snqf0cKnkhA1 IZ3dLri3iITKx1o65wRVm6bwyEv1gnfpvsizzGHfJQrSNDTtgJxk8tmLcbajYzBA OJA1ASn2woSDuMTnZr4Ip/0AwaHMRNa98/nC6iWh1B5iENqmoouEGNm5WcpZQtAi 3eluXgNwlrPBwTw9kDqn0ksDwa2GPbUxCZO+gWzQ886tKa+gz9FXgWJBSmW+dDEB wxY8aO4Pz2r3IrvHVOHlEK44BOh2Jp0DHEK/s82OTf3/++9Qa7jBG9mLDr3NYnoY 7V5Jv8FAirX6vjKFezargA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1754848638; x= 1754935038; bh=CKJHwhQNQy+ROAE4lhfUgG6FBAvwCYWmzuMLynqXNDE=; b=i bFpBTjGb9fCWYdq3sYwnc0SHSZGr0WTjAbLnYO7IqS1tpAd7njgM7Xu0qFrtpP3m gFdnLnit6gx9yjF8AkfMVilRyZ1VEeSQEaVcTMol2DHkYYs6M4myxRUvV0l0UUE4 0Yd46zL8595Ndxs7cyMZEsrAPz75trPsXwlTkt8C+1UyeOlpLnMmUUylTDh9ZCg3 IDuKdeRW4lihW8dScWLFA+LVWbakEISc7Y/7z+0dhDL6lPF4ybEluUcOv1fo1Xr/ pqrbaQaDx8aTZnBXTqKbWFMHmmflnsoFrHpLvg4uPqQ2suyvsYTyk4qAXVeMn2w3 m5njHCjRRezeCmXN3lVhA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtdefgddufedtvdefucetufdoteggodetrf dotffvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfurfetoffkrfgpnffqhgenuceu rghilhhouhhtmecufedttdenucenucfjughrpefhvfevufffkffojghfgggtgfesthekre dtredtjeenucfhrhhomheptehrnhhouhhtucfgnhhgvghlvghnuceorghrnhhouhhtsegs iiiithdrnhgvtheqnecuggftrfgrthhtvghrnheptefhueeugeduueffudfgveeuveetje ehgeetvdevjefghfeltdethfeugeegvedvnecuffhomhgrihhnpehkvghrnhgvlhdrohhr ghdptghouggvsggvrhhgrdhorhhgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrg hmpehmrghilhhfrhhomheprghrnhhouhhtsegsiiiithdrnhgvthdpnhgspghrtghpthht ohepudeipdhmohguvgepshhmthhpohhuthdprhgtphhtthhopehrhigrnheslhgrhhhfrg drgiihiidprhgtphhtthhopegrnhhtohhnhidrrghnthhonhihsehsvggtuhhnvghtrdgt ohhmpdhrtghpthhtoheprghnthhonhihsehphhgvnhhomhgvrdhorhhgpdhrtghpthhtoh eprghsmhgruggvuhhssegtohguvgifrhgvtghkrdhorhhgpdhrtghpthhtohepsghrrghu nhgvrheskhgvrhhnvghlrdhorhhgpdhrtghpthhtohepughhohifvghllhhssehrvgguhh grthdrtghomhdprhgtphhtthhopegvrhhitghvhheskhgvrhhnvghlrdhorhhgpdhrtghp thhtoheplhhinhhugidqfhhsuggvvhgvlhesvhhgvghrrdhkvghrnhgvlhdrohhrghdprh gtphhtthhopehlihhnuhigqdhkvghrnhgvlhesvhhgvghrrdhkvghrnhgvlhdrohhrgh X-ME-Proxy: Feedback-ID: i8a1146c4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 10 Aug 2025 13:57:13 -0400 (EDT) From: Arnout Engelen To: ryan@lahfa.xyz Cc: antony.antony@secunet.com, antony@phenome.org, asmadeus@codewreck.org, brauner@kernel.org, dhowells@redhat.com, ericvh@kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux_oss@crudebyte.com, lucho@ionkov.net, maximilian@mbosch.me, netfs@lists.linux.dev, regressions@lists.linux.dev, sedat.dilek@gmail.com, v9fs@lists.linux.dev Subject: Re: [REGRESSION] 9pfs issues on 6.12-rc1 Date: Sun, 10 Aug 2025 19:57:11 +0200 Message-ID: <20250810175712.3588005-1-arnout@bzzt.net> X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: netfs@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On Fri, 13 Jun 2025 00:24:13 +0200, Ryan Lahfa wrote: > Le Wed, Oct 23, 2024 at 09:38:39PM +0200, Antony Antony a écrit : > > On Wed, Oct 23, 2024 at 11:07:05 +0100, David Howells wrote: > > > Hi Antony, > > > > > > I think the attached should fix it properly rather than working around it as > > > the previous patch did. If you could give it a whirl? > > > > Yes this also fix the crash. > > > > Tested-by: Antony Antony > > I cannot confirm this fixes the crash for me. My reproducer is slightly > more complicated than Max's original one, albeit, still on NixOS and > probably uses 9p more intensively than the automated NixOS testings > workload. I'm seeing a problem in the same area - the symptom is slightly different, but the location seems very similar. I'm also running a NixOS image. Mounting a 9p filesystem in qemu with `cache=readahead`, reading a 12943-byte file, in the guest I do see a 12943-byte file, but only the first 12288 bytes are populated: the rest are zero. This also reproduces (most but not all of the time) on 6.16-rc7, but not on all host machines I've tried. After applying a simplified version of [1] (i.e. [2]), the problem does not reproduce anymore. It seems something in `p9_client_read_once` somehow leaves the iov_iter in an unhealthy state. It would be good to understand exactly what, but I haven't been able to figure that out yet. I have a smallish nix-based reproducer at [3], and a more involved setup with a lot of logging enabled and a convenient way to attach gdb at [4]. You start the VM and then 'cat /repro/default.json' manually, and see if it looks 'truncated'. Interestingly, the file is read in two p9 read calls: one of 12288 bytes and one of 655 bytes. The first read is a zero-copy one, the second is not zero-copy (because it is smaller than 1024). I've also tried with a slightly larger version of the file, that is read as 2 zero-copy reads, and I have not been able to reproduce the problem with that. From my (admittedly limited) understanding the non-zerocopy code path looks fine, though. I hope this is helpful - I'd be happy to keep looking into this further, but any help pointing me in the right direction would be much appreciated :) Kind regards, Arnout [1] https://lore.kernel.org/all/3327438.1729678025@warthog.procyon.org.uk/T/#mc97a248b0f673dff6dc8613b508ca4fd45c4fefe [2] https://codeberg.org/raboof/nextcloud-onlyoffice-test-vm/src/branch/reproducer-with-debugging/kernel-use-copied-iov_iter.patch [3] https://codeberg.org/raboof/nextcloud-onlyoffice-test-vm/src/branch/small-reproducer [4] https://codeberg.org/raboof/nextcloud-onlyoffice-test-vm/src/branch/reproducer-with-debugging