[Virtio-fs] [RFC] Using hard links instead of lo_inode->fd

[Virtio-fs] [RFC] Using hard links instead of lo_inode->fd

[Vivek Goyal]

Hi All,

Max has been trying to solve the problem of not having to keep an O_PATH
fd open for every file (until inode goes away). Lot of inodes can get
cached in guest inode cache and soon we can hit number of max fd limit.

Today Christophe de Dinechin mentioned that can we maintain hardlinks
for these files and open these hardlinks instead whenever we need
lo_inode->fd.

So say there is a notion of "workdir" which is passed to virtiofs where
virtiofsd can create hardlinks. Once inode comes into existence, we
create this link in workdir/<file-link>. Once last reference to inode
is dropped, workdir/<file-link> is removed.

Now any operation which needs the lo_inode->fd, can do operation on
workdir/<file-link>.

How does this proposal sound. What are the issues where. Some of the
which came to surface during discussion with david gilbert are.

1. links can't be created for directories. So this will only help with
  regular files. 

2. There might be a performance penalty. We don't know yet.

3. More overhead of managing workdir by the user/tools.


Anything else? Is this a viable idea even if we can implement only
for regular files.

Thanks
Vivek

Re: [Virtio-fs] [RFC] Using hard links instead of lo_inode->fd

[Miklos Szeredi]

On Thu, Apr 8, 2021 at 7:52 PM Vivek Goyal <vgoyal@redhat.com> wrote:
>
> Hi All,
>
> Max has been trying to solve the problem of not having to keep an O_PATH
> fd open for every file (until inode goes away). Lot of inodes can get
> cached in guest inode cache and soon we can hit number of max fd limit.
>
> Today Christophe de Dinechin mentioned that can we maintain hardlinks
> for these files and open these hardlinks instead whenever we need
> lo_inode->fd.
>
> So say there is a notion of "workdir" which is passed to virtiofs where
> virtiofsd can create hardlinks. Once inode comes into existence, we
> create this link in workdir/<file-link>. Once last reference to inode
> is dropped, workdir/<file-link> is removed.
>
> Now any operation which needs the lo_inode->fd, can do operation on
> workdir/<file-link>.
>
> How does this proposal sound. What are the issues where. Some of the
> which came to surface during discussion with david gilbert are.
>
> 1. links can't be created for directories. So this will only help with
>   regular files.
>
> 2. There might be a performance penalty. We don't know yet.
>
> 3. More overhead of managing workdir by the user/tools.
>
>
> Anything else? Is this a viable idea even if we can implement only
> for regular files.

- nlink offset (this could be corrected)
- needless ctime time update (hard to correct)

I guess this could be a fallback in case the filehandle stuff isn't
working (e.g. because of an old host kernel).

Thanks.
Miklos

Re: [Virtio-fs] [RFC] Using hard links instead of lo_inode->fd

[Max Reitz]

On 08.04.21 19:51, Vivek Goyal wrote:
> Hi All,
> 
> Max has been trying to solve the problem of not having to keep an O_PATH
> fd open for every file (until inode goes away). Lot of inodes can get
> cached in guest inode cache and soon we can hit number of max fd limit.
> 
> Today Christophe de Dinechin mentioned that can we maintain hardlinks
> for these files and open these hardlinks instead whenever we need
> lo_inode->fd.
> 
> So say there is a notion of "workdir" which is passed to virtiofs where
> virtiofsd can create hardlinks. Once inode comes into existence, we
> create this link in workdir/<file-link>. Once last reference to inode
> is dropped, workdir/<file-link> is removed.
> 
> Now any operation which needs the lo_inode->fd, can do operation on
> workdir/<file-link>.
> 
> How does this proposal sound. What are the issues where. Some of the
> which came to surface during discussion with david gilbert are.
> 
> 1. links can't be created for directories. So this will only help with
>    regular files.
> 
> 2. There might be a performance penalty. We don't know yet.
> 
> 3. More overhead of managing workdir by the user/tools.
> 
> 
> Anything else? Is this a viable idea even if we can implement only
> for regular files.

What would happen with submounts?  Wouldn’t we need a workdir per shared 
filesystem then?  (Perhaps not with bind mounts as submounts, I don’t 
know whether hard links work across all bind mounts of some FS.)

Max

Re: [Virtio-fs] [RFC] Using hard links instead of lo_inode->fd

[Vivek Goyal]

On Fri, Apr 09, 2021 at 11:03:47AM +0200, Miklos Szeredi wrote:
> On Thu, Apr 8, 2021 at 7:52 PM Vivek Goyal <vgoyal@redhat.com> wrote:
> >
> > Hi All,
> >
> > Max has been trying to solve the problem of not having to keep an O_PATH
> > fd open for every file (until inode goes away). Lot of inodes can get
> > cached in guest inode cache and soon we can hit number of max fd limit.
> >
> > Today Christophe de Dinechin mentioned that can we maintain hardlinks
> > for these files and open these hardlinks instead whenever we need
> > lo_inode->fd.
> >
> > So say there is a notion of "workdir" which is passed to virtiofs where
> > virtiofsd can create hardlinks. Once inode comes into existence, we
> > create this link in workdir/<file-link>. Once last reference to inode
> > is dropped, workdir/<file-link> is removed.
> >
> > Now any operation which needs the lo_inode->fd, can do operation on
> > workdir/<file-link>.
> >
> > How does this proposal sound. What are the issues where. Some of the
> > which came to surface during discussion with david gilbert are.
> >
> > 1. links can't be created for directories. So this will only help with
> >   regular files.
> >
> > 2. There might be a performance penalty. We don't know yet.
> >
> > 3. More overhead of managing workdir by the user/tools.
> >
> >
> > Anything else? Is this a viable idea even if we can implement only
> > for regular files.
> 
> - nlink offset (this could be corrected)

Oh yes, extra nlink count will be there which will have to be corrected
in getattr().

> - needless ctime time update (hard to correct)

This can pose bigger problems. Sepecially if some operation does not
result in ctime update and it gets updated as a side affect of operating
on this link. In that case a user or a test might complain and that's
sort of hard to fix.  All this ctime/mtime/atime logic is already
pretty complicated to handle.

Thanks
Vivek

> 
> I guess this could be a fallback in case the filehandle stuff isn't
> working (e.g. because of an old host kernel).
> 
> Thanks.
> Miklos
> 

Re: [Virtio-fs] [RFC] Using hard links instead of lo_inode->fd

[Vivek Goyal]

On Fri, Apr 09, 2021 at 11:53:34AM +0200, Max Reitz wrote:
> On 08.04.21 19:51, Vivek Goyal wrote:
> > Hi All,
> > 
> > Max has been trying to solve the problem of not having to keep an O_PATH
> > fd open for every file (until inode goes away). Lot of inodes can get
> > cached in guest inode cache and soon we can hit number of max fd limit.
> > 
> > Today Christophe de Dinechin mentioned that can we maintain hardlinks
> > for these files and open these hardlinks instead whenever we need
> > lo_inode->fd.
> > 
> > So say there is a notion of "workdir" which is passed to virtiofs where
> > virtiofsd can create hardlinks. Once inode comes into existence, we
> > create this link in workdir/<file-link>. Once last reference to inode
> > is dropped, workdir/<file-link> is removed.
> > 
> > Now any operation which needs the lo_inode->fd, can do operation on
> > workdir/<file-link>.
> > 
> > How does this proposal sound. What are the issues where. Some of the
> > which came to surface during discussion with david gilbert are.
> > 
> > 1. links can't be created for directories. So this will only help with
> >    regular files.
> > 
> > 2. There might be a performance penalty. We don't know yet.
> > 
> > 3. More overhead of managing workdir by the user/tools.
> > 
> > 
> > Anything else? Is this a viable idea even if we can implement only
> > for regular files.
> 
> What would happen with submounts?  Wouldn’t we need a workdir per shared
> filesystem then?  (Perhaps not with bind mounts as submounts, I don’t know
> whether hard links work across all bind mounts of some FS.)

That's a good point Max. I don't think hard links can be created
across FS boundaries. So if there are submounts from other filesystems,
then there will have to be workdir on that filesystem, IIUC. Hmm..,
this sounds like a very significant limitation of this proposal.

Thanks
Vivek

Re: [Virtio-fs] [RFC] Using hard links instead of lo_inode->fd

[Dr. David Alan Gilbert]

* Vivek Goyal (vgoyal@redhat.com) wrote:
> Hi All,
> 
> Max has been trying to solve the problem of not having to keep an O_PATH
> fd open for every file (until inode goes away). Lot of inodes can get
> cached in guest inode cache and soon we can hit number of max fd limit.
> 
> Today Christophe de Dinechin mentioned that can we maintain hardlinks
> for these files and open these hardlinks instead whenever we need
> lo_inode->fd.
> 
> So say there is a notion of "workdir" which is passed to virtiofs where
> virtiofsd can create hardlinks. Once inode comes into existence, we
> create this link in workdir/<file-link>. Once last reference to inode
> is dropped, workdir/<file-link> is removed.
> 
> Now any operation which needs the lo_inode->fd, can do operation on
> workdir/<file-link>.
> 
> How does this proposal sound. What are the issues where. Some of the
> which came to surface during discussion with david gilbert are.
> 
> 1. links can't be created for directories. So this will only help with
>   regular files. 
> 
> 2. There might be a performance penalty. We don't know yet.
> 
> 3. More overhead of managing workdir by the user/tools.
> 
> 
> Anything else? Is this a viable idea even if we can implement only
> for regular files.

The other one is that linkat(, AT_EMPTY_PATH) which can create the
link from the existing O_PATH we have, can only be used with
CAP_DAC_READ_SEARCH - which is a shame since otherwise it would seem to
make life easier.

Dave

> Thanks
> Vivek
-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Virtio-fs] [RFC] Using hard links instead of lo_inode->fd

[Vivek Goyal]

On Mon, Apr 12, 2021 at 06:54:52PM +0100, Dr. David Alan Gilbert wrote:
> * Vivek Goyal (vgoyal@redhat.com) wrote:
> > Hi All,
> > 
> > Max has been trying to solve the problem of not having to keep an O_PATH
> > fd open for every file (until inode goes away). Lot of inodes can get
> > cached in guest inode cache and soon we can hit number of max fd limit.
> > 
> > Today Christophe de Dinechin mentioned that can we maintain hardlinks
> > for these files and open these hardlinks instead whenever we need
> > lo_inode->fd.
> > 
> > So say there is a notion of "workdir" which is passed to virtiofs where
> > virtiofsd can create hardlinks. Once inode comes into existence, we
> > create this link in workdir/<file-link>. Once last reference to inode
> > is dropped, workdir/<file-link> is removed.
> > 
> > Now any operation which needs the lo_inode->fd, can do operation on
> > workdir/<file-link>.
> > 
> > How does this proposal sound. What are the issues where. Some of the
> > which came to surface during discussion with david gilbert are.
> > 
> > 1. links can't be created for directories. So this will only help with
> >   regular files. 
> > 
> > 2. There might be a performance penalty. We don't know yet.
> > 
> > 3. More overhead of managing workdir by the user/tools.
> > 
> > 
> > Anything else? Is this a viable idea even if we can implement only
> > for regular files.
> 
> The other one is that linkat(, AT_EMPTY_PATH) which can create the
> link from the existing O_PATH we have, can only be used with
> CAP_DAC_READ_SEARCH - which is a shame since otherwise it would seem to
> make life easier.

But that does not mitigate issues w.r.t submounts (links across
filesytem) and ctime updates?

So only thing this linkat(,AT_EMPTY_PATH) seems to buy is that link
source can be an fd instead of path, right? 

Vivek

Re: [Virtio-fs] [RFC] Using hard links instead of lo_inode->fd

[Dr. David Alan Gilbert]

* Vivek Goyal (vgoyal@redhat.com) wrote:
> On Mon, Apr 12, 2021 at 06:54:52PM +0100, Dr. David Alan Gilbert wrote:
> > * Vivek Goyal (vgoyal@redhat.com) wrote:
> > > Hi All,
> > > 
> > > Max has been trying to solve the problem of not having to keep an O_PATH
> > > fd open for every file (until inode goes away). Lot of inodes can get
> > > cached in guest inode cache and soon we can hit number of max fd limit.
> > > 
> > > Today Christophe de Dinechin mentioned that can we maintain hardlinks
> > > for these files and open these hardlinks instead whenever we need
> > > lo_inode->fd.
> > > 
> > > So say there is a notion of "workdir" which is passed to virtiofs where
> > > virtiofsd can create hardlinks. Once inode comes into existence, we
> > > create this link in workdir/<file-link>. Once last reference to inode
> > > is dropped, workdir/<file-link> is removed.
> > > 
> > > Now any operation which needs the lo_inode->fd, can do operation on
> > > workdir/<file-link>.
> > > 
> > > How does this proposal sound. What are the issues where. Some of the
> > > which came to surface during discussion with david gilbert are.
> > > 
> > > 1. links can't be created for directories. So this will only help with
> > >   regular files. 
> > > 
> > > 2. There might be a performance penalty. We don't know yet.
> > > 
> > > 3. More overhead of managing workdir by the user/tools.
> > > 
> > > 
> > > Anything else? Is this a viable idea even if we can implement only
> > > for regular files.
> > 
> > The other one is that linkat(, AT_EMPTY_PATH) which can create the
> > link from the existing O_PATH we have, can only be used with
> > CAP_DAC_READ_SEARCH - which is a shame since otherwise it would seem to
> > make life easier.
> 
> But that does not mitigate issues w.r.t submounts (links across
> filesytem) and ctime updates?

submounts no, I don't think I understand the ctime changes though.

> So only thing this linkat(,AT_EMPTY_PATH) seems to buy is that link
> source can be an fd instead of path, right? 

Yes, and we already have that fd.

Dave

> Vivek
-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK