From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wm1-f46.google.com (mail-wm1-f46.google.com [209.85.128.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 242DD395AE2 for ; Tue, 13 Oct 2026 09:56:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.46 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791885402; cv=none; b=s73EhVcS0p7C8MwZ8iLxcxShUN72zA5O788IJcS/1vNZcLzd2aD2AzNC4PZwp04VqUjx5Ri4oqsmDJKrDXdV2DmnGjztQhW+3bi5pjN2fi/k2Jcnd663dAC2cuA3/y+hS7U10VLnVxRj5Eastw1MlrXffZ+U0t/+V4MwrcckNOA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791885402; c=relaxed/simple; bh=HFJ4UjRCkhaihkIH92sybsRJA7S5NLo3OntK/t8+1bY=; h=From:To:Cc:Subject:Message-ID:In-Reply-To:References:Date: MIME-Version:Content-Type; b=Vuy60l0uuz9TKE3FTq0DwO5ASFl2vtngAare1bH7EBUWsOqRel2LiVWD1nxJ8oWRLhiagfXaZ9e5gP+XYYJ/TR2oBoXSwH/DnmKFxdeqnBFEqcp8tmn/BntZwFFrgWzN6mlOF+5TfURPsJGJnkt3lOH2nRDeYZkvuOyjeNJAmhc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=innovatrics.com; spf=pass smtp.mailfrom=innovatrics.com; dkim=pass (1024-bit key) header.d=innovatrics.com header.i=@innovatrics.com header.b=f2qTN+IP; arc=none smtp.client-ip=209.85.128.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=innovatrics.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=innovatrics.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=innovatrics.com header.i=@innovatrics.com header.b="f2qTN+IP" Received: by mail-wm1-f46.google.com with SMTP id 5b1f17b1804b1-4a02718da81so21472215e9.1 for ; Tue, 13 Oct 2026 02:56:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=innovatrics.com; s=google; t=1791885399; x=1792490199; darn=vger.kernel.org; h=content-type:mime-version:date:references:in-reply-to:message-id :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to :content-type; bh=St+5KkhKptBQbmhWMvm7sQZJ2JL6wlGqGqKQDJo/9iM=; b=f2qTN+IPbgAf9AFTnqIFnA0+EFd9chhkqWbo9ujAYlUoIIB6b94DVLEFvuMMx4MDht rDU7HE0clCOb2uv2Ax05WtETaT+AAYaHDmmY2vekq4etv11VynKz/Uhw/cEtYcYLnVSJ +0JDljA8j7m5b9RdH1PAEZPDC3Xf6DttepzLE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1791885399; x=1792490199; h=content-type:mime-version:date:references:in-reply-to:message-id :subject:cc:to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to:content-type; bh=St+5KkhKptBQbmhWMvm7sQZJ2JL6wlGqGqKQDJo/9iM=; b=S0HZ0IDCBqLgFrnWag7EiWdTQJFT9XXZijRRZ4GBa/5+P0wstaywJDEw1qvGqDgNMs vsu1O1ceK3mybl0Qj+y4TKQhFsFUWP34yXJnT9siSoBbI4wv4Mma+83k/RI02hi/Wo0A Yw7871VYn/g1vKphDma5nhC72wn+1f1VNXsXCLFyYkLW7sjYBss6vYzjoxHigbquozdO ExdQMtcs91uNYdqI9+JBeH3p3N8F38KNMTcB6/oFmHlvHXnMnhrWcMDd1jnaPG+Z0tRH kQ7qRJvN+OtRhBWnPNKx6iVnybCgYjFx9Vp3xcKfW/GnM+G8IFHE1NLqqWwiEzTZUSjS uz7g== X-Forwarded-Encrypted: i=1; AKwUvByYHkah7219IcMksjZP6uDwVogDKsIXIfsTinxSuiVFcbQhT/DULFjnLwcWurERS9h9XFsYqyhWwFlXxbru@vger.kernel.org X-Gm-Message-State: AFq9FYJvX8yQtMg7Bh5bgSFStZdnOYST4uvAOoEP0jX+mmapP7rkqYpo 2c1AWrb+RM//qtzMAoreOfBV9sNhuq+cVyQXt9kCtebSZaOVwGs04Ero2+WCz8L9Hg== X-Gm-Gg: AYBFou0N+AGYKJqeXOlwQ6X+ku8PBGPjckeHNduxq3yCzIxCQKtoXEyXO+NG/3zIHAT bQAJuKPUrqz1ItwtCLgrJkAjN2wngYT7KOjynmmwMLFKGYbPKF2DYLDJZUwwIWyz1sCZRWGhvJK WZuZaIe9KxKAFSEQPupUY4bUghVkO9nVJSHdGNTQnEIJ7ly2zTLaqcKT/om/ePa3/ipDDPMk5Ck pezw44CnxkqrLtIJKHbcFBEuDyTPOIKNGTWt9IQkbxXH6xEBaYR2Jxnu2lMcZAudsrLcJL1A+07 ccDHzgryPq4i8yURwn/TM1AIaHyA3GCFWqzRfmmCYEqwrdTe+cx1xy7eFnm5yi1M2QxGMRTapnA WRJCM9Ssxn6pLSiubCf3bCBmVtMTkYLjm2kLHbwuQ9z8kzj0GJqnJhYueiysSwIv5WskPRSMbuH O019J/9LC/5vQwb8W+8QKelQS0vv66iFczN1+MBOHkX802UgBGykVp5JIoPxoHea0Wn44dIIbyg Zj4jINnp3Z143vmbyanYaEvZ8FEXqsw0/VIvrKsKRT/d/vLKvatyyywyOLTQQh0xWWJGg4N6bQh 7RzU99VYjyYHI8ggpWkxrT3rZPT8tNI= X-Received: by 2002:a05:600c:1c07:b0:4a1:b044:9ccd with SMTP id 5b1f17b1804b1-4a1b0449d00mr3292955e9.28.1791885399299; Tue, 13 Oct 2026 02:56:39 -0700 (PDT) Received: from [10.11.72.5] (bip-static-148.213-81-147.telecom.sk. [213.81.147.148]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4a19f249692sm289830295e9.3.2026.10.13.02.56.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Oct 2026 02:56:38 -0700 (PDT) From: Petr Chmelar To: David Howells Cc: Christian Brauner , Paulo Alcantara , Matthew Wilcox , Namjae Jeon , Marc Dionne , Stefan Metzmacher , Eric Van Hensbergen , Dominique Martinet , Ilya Dryomov , netfs@lists.linux.dev, linux-afs@lists.infradead.org, linux-cifs@vger.kernel.org, linux-nfs@vger.kernel.org, ceph-devel@vger.kernel.org, v9fs@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v12 4/5] cachefiles: Don't rely on backing fs storage map for most use cases Message-ID: <20261013.cachefiles-partial-zeros-trace.pchmelar@innovatrics.com> In-Reply-To: <1655268.1791878098@warthog.procyon.org.uk> References: <20260910220242.2165023-1-dhowells@redhat.com> <20260910220242.2165023-5-dhowells@redhat.com> <20261011.cachefiles-partial-zeros.pchmelar@innovatrics.com> <1655268.1791878098@warthog.procyon.org.uk> Date: Tue, 13 Oct 2026 09:56:37 +0000 Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" David Howells wrote: > Unfortunately, I can't reproduce this. Can you get me some tracing? Thanks for looking. The trace shows reads served from the cache over ranges that were never written to it; details below. I think the reason you don't see it is how the file gets into the cache, which I should have said in the report. In your run, fio lays the file out through the mount, so the client fills its cache sequentially and the object ends up CONTENT_ALL. In ours, the client never writes the file: it is written on the server side beforehand, and the client's cache fills only as the client reads it, at random offsets, so the object stays BACKFS_MAP. That is how our training nodes use the cache: they read data sets that they didn't write, mostly at random offsets. I reran it on 9f8b64f66 over 9p with your tracepoints enabled, with the same result: 936 of 4096 64 KiB blocks wrong, 825 of them zeros, first wrong blocks 10, 13, 15, 24, 26 and 27. I'm sending you the trace off-list; it is 790 KB as bzip2. What the trace shows: - After the random reads the object is set to BACKFS_MAP: cachefiles_coherency: o=00000001 SET ok B=c4 oz=10000000 c=3 After dropping the page cache it is looked up again (o=00000002 OK, c=3) and set to c=4. - Before the full read, WRIT TERM subrequests cover 2048 of the 4096 64 KiB blocks, as expected for half the file. - During the full read, READ subrequests from the cache cover 2143 blocks, and 936 of those never had a WRIT: the same count as the wrong blocks, starting with the same blocks, 10, 13, 15, 24, 26 and 27. For example this one read: R=0000285a[1] READ TERM f=110 s=7c000 7c000/7c000 s=7 e=0 covers 0xa0000-0xb0000, 0xd0000-0xe0000 and 0xf0000-0xf8000, and no write ever reached those ranges. The writes around it were: R=00002291[3] WRIT TERM f=100 s=70000 20000/20000 s=6 e=0 R=000024dc[1] WRIT TERM f=100 s=90000 10000/10000 s=16 e=0 R=00002291[4] WRIT TERM f=100 s=b0000 20000/20000 s=9 e=0 R=000024dc[2] WRIT TERM f=100 s=e0000 10000/10000 s=3 e=0 So for the BACKFS_MAP object, the occupancy the read is based on treats the holes between those writes as data. > One further question occurs to me: does the entire 256M file fit in the > test VM's memory? Yes. The VM has 7946 MiB of memory with 7498 MiB available, and the page cache is dropped before the full read. The setup, for reference: - 9f8b64f66 with netfs, fscache, cachefiles, 9p and 9p fscache built in; QEMU without KVM, 4 vCPUs, 8 GiB RAM - 9p: QEMU virtfs local, security_model=none, the host directory on XFS; mounted with trans=virtio,version=9p2000.L,cache=fscache,msize=1048576 - part.bin: 256 MiB from /dev/urandom, written on the host before the mount - cache: XFS on a loop device in the guest, cachefilesd 0.10.10 The trace analysis was done with LLM assistance. Thanks, Petr