Re: [LSF/MM/BPF TOPIC] How to make disconnected operation work?

[Jeff Layton <jlayton@redhat.com>]

On Mon, 2019-12-09 at 14:46 +0000, David Howells wrote:
> I've been rewriting fscache and cachefiles to massively simplify it and make
> use of the kiocb interface to do direct-I/O to/from the netfs's pages which
> didn't exist when I first did this.
> 
> 	https://lore.kernel.org/lkml/24942.1573667720@warthog.procyon.org.uk/
> 	https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git/log/?h=fscache-iter
> 
> I'm getting towards the point where it's working and able to do basic caching
> once again.  So now I've been thinking about what it'd take to support
> disconnected operation.  Here's a list of things that I think need to be
> considered or dealt with:
> 

I'm quite interested in this too. I see that you've already given a lot
of thought to potential interfaces here. I think we'll end up having to
add a fair number of new interfaces to make something like this work.

>  (1) Making sure the working set is present in the cache.
> 
>      - Userspace (find/cat/tar)
>      - Splice netfs -> cache
>      - Metadata storage (e.g. directories)
>      - Permissions caching
> 
>  (2) Making sure the working set doesn't get culled.
> 
>      - Pinning API (cachectl() syscall?)
>      - Allow culling to be disabled entirely on a cache
>      - Per-fs/per-dir config
> 
>  (3) Switching into/out of disconnected mode.
> 
>      - Manual, automatic
>      - On what granularity?
>        - Entirety of fs (eg. all nfs)
>        - By logical unit (server, volume, cell, share)
>
>  (4) Local changes in disconnected mode.
> 
>      - Journal
>      - File identifier allocation

Yep, necessary if you want to allow disconnected creates. By coincidence
I'm working an (experimental) patchset now to add async create support
to kcephfs, and part of that involves delegating out ranges of inode
numbers. I may have some experience to report with it by the time LSF
rolls around.

>      - statx flag to indicate provisional nature of info
>      - New error codes
> 	- EDISCONNECTED - Op not available in disconnected mode
> 	- EDISCONDATA - Data not available in disconnected mode
> 	- EDISCONPERM - Permission cannot be checked in disconnected mode
> 	- EDISCONFULL - Disconnected mode cache full
>      - SIGIO support?
> 
>  (5) Reconnection.
> 
>      - Proactive or JIT synchronisation
>        - Authentication
>      - Conflict detection and resolution
> 	 - ECONFLICTED - Disconnected mode resolution failed

ECONFLICTED sort of implies that reconnection will be manual. If it
happens automagically in the background you'll have no way to report
such errors.

Also, you'll need some mechanism to know what inodes are conflicted.
This is the real difficult part of this problem, IMO.


>      - Journal replay
>      - Directory 'diffing' to find remote deletions
>      - Symlink and other non-regular file comparison
> 
>  (6) Conflict resolution.
> 
>      - Automatic where possible
>        - Just create/remove new non-regular files if possible
>        - How to handle permission differences?
>      - How to let userspace access conflicts?
>        - Move local copy to 'lost+found'-like directory
>        	 - Might not have been completely downloaded
>        - New open() flags?
>        	 - O_SERVER_VARIANT, O_CLIENT_VARIANT, O_RESOLVED_VARIANT
>        - fcntl() to switch variants?
> 

Again, conflict resolution is the difficult part. Maybe the right
solution is to look at snapshotting-style interfaces -- i.e., handle a
disconnected mount sort of like you would a writable snapshot. Do any
(local) fs' currently offer writable snapshots, btw?

>  (7) GUI integration.
> 
>      - Entering/exiting disconnected mode notification/switches.
>      - Resolution required notification.
>      - Cache getting full notification.
> 
> Can anyone think of any more considerations?  What do you think of the
> proposed error codes and open flags?  Is that the best way to do this?
> 
> David
> 

-- 
Jeff Layton <jlayton@redhat.com>