odd sysfs find behavior, was: Re: [PATCH v6 00/10] Implement NVMe Namespace Descriptor Identification

odd sysfs find behavior, was: Re: [PATCH v6 00/10] Implement NVMe Namespace Descriptor Identification

[Christoph Hellwig]

On Fri, Jun 16, 2017 at 10:28:28PM +0900, Linus Torvalds wrote:
> Hmm. The *traditional* reason for this particular 'find' oddity is
> that find has an optimization which will look at the nlink count of a
> directory to decide how many subdirectories it can have.

It looks like sysfs maintains i_nlink properly as far as I can tell.
But I noticed another things that's even more weird:

root@testvm:~/nvmetcli# find /sys -name uuid
root@testvm:~/nvmetcli# find /sys -name uuid
/sys/devices/virtual/nvme-fabrics/ctl/nvme2/nvme2n1/uuid

so just repeating the find a second time makes it work!  Looks like
there is some sort of lazy population scheme in sysfs..

Re: odd sysfs find behavior, was: Re: [PATCH v6 00/10] Implement NVMe Namespace Descriptor Identification

[Greg Kroah-Hartman]

On Sat, Jun 17, 2017 at 02:08:14PM +0200, Christoph Hellwig wrote:
> On Fri, Jun 16, 2017 at 10:28:28PM +0900, Linus Torvalds wrote:
> > Hmm. The *traditional* reason for this particular 'find' oddity is
> > that find has an optimization which will look at the nlink count of a
> > directory to decide how many subdirectories it can have.
> 
> It looks like sysfs maintains i_nlink properly as far as I can tell.
> But I noticed another things that's even more weird:
> 
> root@testvm:~/nvmetcli# find /sys -name uuid
> root@testvm:~/nvmetcli# find /sys -name uuid
> /sys/devices/virtual/nvme-fabrics/ctl/nvme2/nvme2n1/uuid
> 
> so just repeating the find a second time makes it work!  Looks like
> there is some sort of lazy population scheme in sysfs..

There is a lazy population scheme in sysfs, but that means that things
are only populated in memory when we actually look for them, not the
second time around only :)

I just tried this on my laptop which does have a nvme block device in
it, and it worked just fine both times.

As Linus said, a strace would be great.

thanks,

greg k-h

Re: odd sysfs find behavior, was: Re: [PATCH v6 00/10] Implement NVMe Namespace Descriptor Identification

[Christoph Hellwig]

On Sun, Jun 18, 2017 at 09:12:50AM +0800, Greg Kroah-Hartman wrote:
> > 
> > root@testvm:~/nvmetcli# find /sys -name uuid
> > root@testvm:~/nvmetcli# find /sys -name uuid
> > /sys/devices/virtual/nvme-fabrics/ctl/nvme2/nvme2n1/uuid
> > 
> > so just repeating the find a second time makes it work!  Looks like
> > there is some sort of lazy population scheme in sysfs..
> 
> There is a lazy population scheme in sysfs, but that means that things
> are only populated in memory when we actually look for them, not the
> second time around only :)
> 
> I just tried this on my laptop which does have a nvme block device in
> it, and it worked just fine both times.

Note that the search is for an attribute in the sysfs dir, but trying
to search for the device it doesn't make any difference.

> As Linus said, a strace would be great.

Both straces attached (gzipped due to size).  Note that this on a system
with four NVMe devices, two of which are nvme-loop devices that have the
uuid attribute.

Re: odd sysfs find behavior, was: Re: [PATCH v6 00/10] Implement NVMe Namespace Descriptor Identification

[Tejun Heo]

Hello, Christoph.

On Sun, Jun 18, 2017 at 09:33:05AM +0200, Christoph Hellwig wrote:
> > Both straces attached (gzipped due to size).  Note that this on a system
> > with four NVMe devices, two of which are nvme-loop devices that have the
> > uuid attribute.
> 
> And now really attached..

If you "cd /sys/kernel/debug/tracing; cd -" before doing the find,
does it succeed the first time?  It could be that the on-demand
mounting of tracefs is giving incoherent results across the mounting
and confusing find.

Thanks.

-- 
tejun

Re: odd sysfs find behavior, was: Re: [PATCH v6 00/10] Implement NVMe Namespace Descriptor Identification

[Tejun Heo]

On Sun, Jun 18, 2017 at 06:20:20AM -0400, Tejun Heo wrote:
> Hello, Christoph.
> 
> On Sun, Jun 18, 2017 at 09:33:05AM +0200, Christoph Hellwig wrote:
> > > Both straces attached (gzipped due to size).  Note that this on a system
> > > with four NVMe devices, two of which are nvme-loop devices that have the
> > > uuid attribute.
> > 
> > And now really attached..
> 
> If you "cd /sys/kernel/debug/tracing; cd -" before doing the find,
> does it succeed the first time?  It could be that the on-demand
> mounting of tracefs is giving incoherent results across the mounting
> and confusing find.

Before tracefs is automounted

    File: '/sys/kernel/debug/tracing'
    Size: 0               Blocks: 0          IO Block: 4096   directory
  Device: 8h/8d   Inode: 2054        Links: 3
  Access: (0555/dr-xr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
  Access: 2017-06-18 03:21:26.241000000 -0700
  Modify: 2017-06-18 03:21:26.241000000 -0700
  Change: 2017-06-18 03:21:26.241000000 -0700
   Birth: -

and after

    File: '/sys/kernel/debug/tracing/'
    Size: 0               Blocks: 0          IO Block: 4096   directory
  Device: 9h/9d   Inode: 1           Links: 6
  Access: (0700/drwx------)  Uid: (    0/    root)   Gid: (    0/    root)
  Access: 2017-06-18 03:21:26.241000000 -0700
  Modify: 2017-06-18 03:21:26.241000000 -0700
  Change: 2017-06-18 03:21:26.241000000 -0700
   Birth: -

and it gets mounted while the first find is iterating through it.

Thanks.

-- 
tejun

Re: odd sysfs find behavior, was: Re: [PATCH v6 00/10] Implement NVMe Namespace Descriptor Identification

[Christoph Hellwig]

On Sun, Jun 18, 2017 at 06:20:20AM -0400, Tejun Heo wrote:
> If you "cd /sys/kernel/debug/tracing; cd -" before doing the find,
> does it succeed the first time?  It could be that the on-demand
> mounting of tracefs is giving incoherent results across the mounting
> and confusing find.

Yes, that works fine:

root@testvm:~# cd /sys/kernel/debug/tracing; cd -
/root
root@testvm:~# find /sys -name uuid
/sys/devices/virtual/nvme-fabrics/ctl/nvme3/nvme3n1/uuid
/sys/devices/virtual/nvme-fabrics/ctl/nvme2/nvme2n1/uuid

Re: odd sysfs find behavior, was: Re: [PATCH v6 00/10] Implement NVMe Namespace Descriptor Identification

[Tejun Heo]

Hello,

On Sun, Jun 18, 2017 at 03:43:36PM +0200, Christoph Hellwig wrote:
> On Sun, Jun 18, 2017 at 06:20:20AM -0400, Tejun Heo wrote:
> > If you "cd /sys/kernel/debug/tracing; cd -" before doing the find,
> > does it succeed the first time?  It could be that the on-demand
> > mounting of tracefs is giving incoherent results across the mounting
> > and confusing find.
> 
> Yes, that works fine:
> 
> root@testvm:~# cd /sys/kernel/debug/tracing; cd -
> /root
> root@testvm:~# find /sys -name uuid
> /sys/devices/virtual/nvme-fabrics/ctl/nvme3/nvme3n1/uuid
> /sys/devices/virtual/nvme-fabrics/ctl/nvme2/nvme2n1/uuid

So, looks like this is the debugfs automount mechanism changing nlinks
across automounts and confusing find in the process.  Greg?

Thanks.

-- 
tejun

Re: odd sysfs find behavior, was: Re: [PATCH v6 00/10] Implement NVMe Namespace Descriptor Identification

[Greg Kroah-Hartman]

On Mon, Jun 19, 2017 at 01:59:34PM -0400, Tejun Heo wrote:
> Hello,
> 
> On Sun, Jun 18, 2017 at 03:43:36PM +0200, Christoph Hellwig wrote:
> > On Sun, Jun 18, 2017 at 06:20:20AM -0400, Tejun Heo wrote:
> > > If you "cd /sys/kernel/debug/tracing; cd -" before doing the find,
> > > does it succeed the first time?  It could be that the on-demand
> > > mounting of tracefs is giving incoherent results across the mounting
> > > and confusing find.
> > 
> > Yes, that works fine:
> > 
> > root@testvm:~# cd /sys/kernel/debug/tracing; cd -
> > /root
> > root@testvm:~# find /sys -name uuid
> > /sys/devices/virtual/nvme-fabrics/ctl/nvme3/nvme3n1/uuid
> > /sys/devices/virtual/nvme-fabrics/ctl/nvme2/nvme2n1/uuid
> 
> So, looks like this is the debugfs automount mechanism changing nlinks
> across automounts and confusing find in the process.  Greg?

That's odd, is find racing with itself here?  I don't really know how to
do anything about this in sysfs, as this should happen for any other
filesystem, right?

thanks,

greg k-h

Re: odd sysfs find behavior, was: Re: [PATCH v6 00/10] Implement NVMe Namespace Descriptor Identification

[Tejun Heo]

Hello,

On Wed, Jun 21, 2017 at 10:48:26PM +0800, Greg Kroah-Hartman wrote:
> On Mon, Jun 19, 2017 at 01:59:34PM -0400, Tejun Heo wrote:
> > So, looks like this is the debugfs automount mechanism changing nlinks
> > across automounts and confusing find in the process.  Greg?
> 
> That's odd, is find racing with itself here?  I don't really know how to
> do anything about this in sysfs, as this should happen for any other
> filesystem, right?

I don't actually know why find gets confused over nlinks changing
underneath it.  Can't we also automount on the first stat(2) call?

Thanks.

-- 
tejun

Re: odd sysfs find behavior, was: Re: [PATCH v6 00/10] Implement NVMe Namespace Descriptor Identification

[Linus Torvalds]

On Mon, Jun 19, 2017 at 10:59 AM, Tejun Heo <tj@kernel.org> wrote:
>
> So, looks like this is the debugfs automount mechanism changing nlinks
> across automounts and confusing find in the process.  Greg?

Btw, you should be able to use the "-noleaf" option to find to disable
the whole "count nlink of directories" logic.

So does

    find /sys -noleaf -name uuid

work reliably whether that automount thing has triggered or not?

That said, I don't understand why the automount of debugfs should
matter. It's not even under the same hierarchy as any of the uuid
files, unless I'm confused. So the nlink on *those* directories
shouldn't even matter. Very odd.

                   Linus