Re: [PATCH v12 4/5] cachefiles: Don't rely on backing fs storage map for most use cases

[Petr Chmelar <petr.chmelar@innovatrics.com>]

David Howells <dhowells@redhat.com> wrote:

> Unfortunately, I can't reproduce this.  Can you get me some tracing?

Thanks for looking.  The trace shows reads served from the cache over
ranges that were never written to it; details below.

I think the reason you don't see it is how the file gets into the cache,
which I should have said in the report.  In your run, fio lays the file
out through the mount, so the client fills its cache sequentially and the
object ends up CONTENT_ALL.  In ours, the client never writes the file:
it is written on the server side beforehand, and the client's cache fills
only as the client reads it, at random offsets, so the object stays
BACKFS_MAP.  That is how our training nodes use the cache: they read data
sets that they didn't write, mostly at random offsets.

I reran it on 9f8b64f66 over 9p with your tracepoints enabled, with the
same result: 936 of 4096 64 KiB blocks wrong, 825 of them zeros, first
wrong blocks 10, 13, 15, 24, 26 and 27.  I'm sending you the trace
off-list; it is 790 KB as bzip2.

What the trace shows:

 - After the random reads the object is set to BACKFS_MAP:

     cachefiles_coherency: o=00000001 SET ok   B=c4 oz=10000000 c=3

   After dropping the page cache it is looked up again (o=00000002 OK,
   c=3) and set to c=4.

 - Before the full read, WRIT TERM subrequests cover 2048 of the 4096
   64 KiB blocks, as expected for half the file.

 - During the full read, READ subrequests from the cache cover 2143
   blocks, and 936 of those never had a WRIT: the same count as the wrong
   blocks, starting with the same blocks, 10, 13, 15, 24, 26 and 27.
   For example this one read:

     R=0000285a[1] READ TERM  f=110 s=7c000 7c000/7c000 s=7 e=0

   covers 0xa0000-0xb0000, 0xd0000-0xe0000 and 0xf0000-0xf8000, and no
   write ever reached those ranges.  The writes around it were:

     R=00002291[3] WRIT TERM  f=100 s=70000 20000/20000 s=6 e=0
     R=000024dc[1] WRIT TERM  f=100 s=90000 10000/10000 s=16 e=0
     R=00002291[4] WRIT TERM  f=100 s=b0000 20000/20000 s=9 e=0
     R=000024dc[2] WRIT TERM  f=100 s=e0000 10000/10000 s=3 e=0

So for the BACKFS_MAP object, the occupancy the read is based on treats
the holes between those writes as data.

> One further question occurs to me: does the entire 256M file fit in the
> test VM's memory?

Yes.  The VM has 7946 MiB of memory with 7498 MiB available, and the page
cache is dropped before the full read.

The setup, for reference:

 - 9f8b64f66 with netfs, fscache, cachefiles, 9p and 9p fscache built in;
   QEMU without KVM, 4 vCPUs, 8 GiB RAM
 - 9p: QEMU virtfs local, security_model=none, the host directory on XFS;
   mounted with trans=virtio,version=9p2000.L,cache=fscache,msize=1048576
 - part.bin: 256 MiB from /dev/urandom, written on the host before the
   mount
 - cache: XFS on a loop device in the guest, cachefilesd 0.10.10

The trace analysis was done with LLM assistance.

Thanks,
Petr