VFS regression: commit aba809cf0944 breaks MNT_SHRINKABLE automounted partitions

VFS regression: commit aba809cf0944 breaks MNT_SHRINKABLE automounted partitions

[Trond Myklebust]

Hi Al,

Our internal testing shows that commit aba809cf0944 (namespace.c: get
rid of mnt_ghosts) appears to break the NFS automounter functionality.
When unmounting, the user now gets a permanent -EBUSY, presumably due
to a refcounting issue with the patch.

The reproducer is as follows:

On the server, set up an export of the form:

% showmount -e 192.168.214.128
Export list for 192.168.214.128:
/export *

On the client:

[fedora19@~]$sudo mount 192.168.214.128:/ /mnt/
[fedora19@~]$df
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/fedora-root   38G   15G   24G  38% /
devtmpfs                 229M     0  229M   0% /dev
tmpfs                    242M   80K  242M   1% /dev/shm
tmpfs                    242M  1.5M  241M   1% /run
tmpfs                    242M     0  242M   0% /sys/fs/cgroup
tmpfs                    242M  4.0K  242M   1% /tmp
/dev/sda1                1.5G  248M  1.3G  17% /boot
192.168.214.128:/         28G   25G  1.1G  96% /mnt
[fedora19@~]$ls /mnt/export/
[fedora19@~]$df
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/fedora-root   38G   15G   24G  38% /
devtmpfs                 229M     0  229M   0% /dev
tmpfs                    242M   80K  242M   1% /dev/shm
tmpfs                    242M  1.5M  241M   1% /run
tmpfs                    242M     0  242M   0% /sys/fs/cgroup
tmpfs                    242M  4.0K  242M   1% /tmp
/dev/sda1                1.5G  248M  1.3G  17% /boot
192.168.214.128:/         28G   25G  1.1G  96% /mnt
192.168.214.128:/export   28G   25G  1.1G  96% /mnt/export
[fedora19@~]$sudo umount /mnt
umount.nfs4: /mnt: device is busy
[fedora19@~]$df
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/fedora-root   38G   15G   24G  38% /
devtmpfs                 229M     0  229M   0% /dev
tmpfs                    242M   80K  242M   1% /dev/shm
tmpfs                    242M  1.5M  241M   1% /run
tmpfs                    242M     0  242M   0% /sys/fs/cgroup
tmpfs                    242M  4.0K  242M   1% /tmp
/dev/sda1                1.5G  248M  1.3G  17% /boot
192.168.214.128:/         28G   25G  1.1G  96% /mnt

Reverting commit aba809cf0944 fixes the problem for us on a 3.14.15
client kernel.
(Thanks to Tao Peng for providing the above reproducer and for bisecting.)

Cheers
  Trond
-- 
Trond Myklebust

Linux NFS client maintainer, PrimaryData

trond.myklebust@primarydata.com

Re: VFS regression: commit aba809cf0944 breaks MNT_SHRINKABLE automounted partitions

[Al Viro]

On Fri, Aug 29, 2014 at 12:27:53PM -0400, Trond Myklebust wrote:
> Hi Al,
> 
> Our internal testing shows that commit aba809cf0944 (namespace.c: get
> rid of mnt_ghosts) appears to break the NFS automounter functionality.
> When unmounting, the user now gets a permanent -EBUSY, presumably due
> to a refcounting issue with the patch.
> 
> The reproducer is as follows:
> 
> On the server, set up an export of the form:
> 
> % showmount -e 192.168.214.128
> Export list for 192.168.214.128:
> /export *
> 
> On the client:
> 
> [fedora19@~]$sudo mount 192.168.214.128:/ /mnt/
> [fedora19@~]$df
> Filesystem               Size  Used Avail Use% Mounted on
> /dev/mapper/fedora-root   38G   15G   24G  38% /
> devtmpfs                 229M     0  229M   0% /dev
> tmpfs                    242M   80K  242M   1% /dev/shm
> tmpfs                    242M  1.5M  241M   1% /run
> tmpfs                    242M     0  242M   0% /sys/fs/cgroup
> tmpfs                    242M  4.0K  242M   1% /tmp
> /dev/sda1                1.5G  248M  1.3G  17% /boot
> 192.168.214.128:/         28G   25G  1.1G  96% /mnt
> [fedora19@~]$ls /mnt/export/
> [fedora19@~]$df
> Filesystem               Size  Used Avail Use% Mounted on
> /dev/mapper/fedora-root   38G   15G   24G  38% /
> devtmpfs                 229M     0  229M   0% /dev
> tmpfs                    242M   80K  242M   1% /dev/shm
> tmpfs                    242M  1.5M  241M   1% /run
> tmpfs                    242M     0  242M   0% /sys/fs/cgroup
> tmpfs                    242M  4.0K  242M   1% /tmp
> /dev/sda1                1.5G  248M  1.3G  17% /boot
> 192.168.214.128:/         28G   25G  1.1G  96% /mnt
> 192.168.214.128:/export   28G   25G  1.1G  96% /mnt/export
> [fedora19@~]$sudo umount /mnt
> umount.nfs4: /mnt: device is busy
> [fedora19@~]$df
> Filesystem               Size  Used Avail Use% Mounted on
> /dev/mapper/fedora-root   38G   15G   24G  38% /
> devtmpfs                 229M     0  229M   0% /dev
> tmpfs                    242M   80K  242M   1% /dev/shm
> tmpfs                    242M  1.5M  241M   1% /run
> tmpfs                    242M     0  242M   0% /sys/fs/cgroup
> tmpfs                    242M  4.0K  242M   1% /tmp
> /dev/sda1                1.5G  248M  1.3G  17% /boot
> 192.168.214.128:/         28G   25G  1.1G  96% /mnt
> 
> Reverting commit aba809cf0944 fixes the problem for us on a 3.14.15
> client kernel.

Can't reproduce on a debian client with 3.12 kernel (which contains the
commit in question); it'll take a bit to get Fedora setup - I have it as KVM
image, but it's sandboxed to hell and back and certainly not allowed to play
with NFS traffic...

In the meanwhile, if you have a reproducer setup...  Could you verify that
fuser -m /mnt does come empty?  Another thing: does mount --make-rprivate /
done before that umount /mnt change behaviour?  IOW, I wonder if it could
be affected by shared-subtree setup; that might be the relevant difference
between the working box and F19 one in this case.  Could you at least post
the contents of /proc/*/mountinfo from all namespaces?  Output of
grep . `md5sum /proc/*/mountinfo|sort|uniq -w32|awk '{print $2}'`
would do...

Re: VFS regression: commit aba809cf0944 breaks MNT_SHRINKABLE automounted partitions

[Trond Myklebust]

[-- Attachment #1: Type: text/plain, Size: 6051 bytes --]

On Fri, Aug 29, 2014 at 4:58 PM, Al Viro <viro@zeniv.linux.org.uk> wrote:
> On Fri, Aug 29, 2014 at 12:27:53PM -0400, Trond Myklebust wrote:
>> Hi Al,
>>
>> Our internal testing shows that commit aba809cf0944 (namespace.c: get
>> rid of mnt_ghosts) appears to break the NFS automounter functionality.
>> When unmounting, the user now gets a permanent -EBUSY, presumably due
>> to a refcounting issue with the patch.
>>
>> The reproducer is as follows:
>>
>> On the server, set up an export of the form:
>>
>> % showmount -e 192.168.214.128
>> Export list for 192.168.214.128:
>> /export *
>>
>> On the client:
>>
>> [fedora19@~]$sudo mount 192.168.214.128:/ /mnt/
>> [fedora19@~]$df
>> Filesystem               Size  Used Avail Use% Mounted on
>> /dev/mapper/fedora-root   38G   15G   24G  38% /
>> devtmpfs                 229M     0  229M   0% /dev
>> tmpfs                    242M   80K  242M   1% /dev/shm
>> tmpfs                    242M  1.5M  241M   1% /run
>> tmpfs                    242M     0  242M   0% /sys/fs/cgroup
>> tmpfs                    242M  4.0K  242M   1% /tmp
>> /dev/sda1                1.5G  248M  1.3G  17% /boot
>> 192.168.214.128:/         28G   25G  1.1G  96% /mnt
>> [fedora19@~]$ls /mnt/export/
>> [fedora19@~]$df
>> Filesystem               Size  Used Avail Use% Mounted on
>> /dev/mapper/fedora-root   38G   15G   24G  38% /
>> devtmpfs                 229M     0  229M   0% /dev
>> tmpfs                    242M   80K  242M   1% /dev/shm
>> tmpfs                    242M  1.5M  241M   1% /run
>> tmpfs                    242M     0  242M   0% /sys/fs/cgroup
>> tmpfs                    242M  4.0K  242M   1% /tmp
>> /dev/sda1                1.5G  248M  1.3G  17% /boot
>> 192.168.214.128:/         28G   25G  1.1G  96% /mnt
>> 192.168.214.128:/export   28G   25G  1.1G  96% /mnt/export
>> [fedora19@~]$sudo umount /mnt
>> umount.nfs4: /mnt: device is busy
>> [fedora19@~]$df
>> Filesystem               Size  Used Avail Use% Mounted on
>> /dev/mapper/fedora-root   38G   15G   24G  38% /
>> devtmpfs                 229M     0  229M   0% /dev
>> tmpfs                    242M   80K  242M   1% /dev/shm
>> tmpfs                    242M  1.5M  241M   1% /run
>> tmpfs                    242M     0  242M   0% /sys/fs/cgroup
>> tmpfs                    242M  4.0K  242M   1% /tmp
>> /dev/sda1                1.5G  248M  1.3G  17% /boot
>> 192.168.214.128:/         28G   25G  1.1G  96% /mnt
>>
>> Reverting commit aba809cf0944 fixes the problem for us on a 3.14.15
>> client kernel.
>
> Can't reproduce on a debian client with 3.12 kernel (which contains the
> commit in question); it'll take a bit to get Fedora setup - I have it as KVM
> image, but it's sandboxed to hell and back and certainly not allowed to play
> with NFS traffic...
>
> In the meanwhile, if you have a reproducer setup...  Could you verify that
> fuser -m /mnt does come empty?  Another thing: does mount --make-rprivate /
> done before that umount /mnt change behaviour?  IOW, I wonder if it could
> be affected by shared-subtree setup; that might be the relevant difference
> between the working box and F19 one in this case.  Could you at least post
> the contents of /proc/*/mountinfo from all namespaces?  Output of
> grep . `md5sum /proc/*/mountinfo|sort|uniq -w32|awk '{print $2}'`
> would do...

Note that the issue happens with NFSv4 or NFSv4.1 (not NFSv3). The
following was done on a Fedora 20 kernel (3.15.10) but the results are
more or less the same:

[root@vmw-test-1 ~]# showmount -e vmw-test-2
Export list for vmw-test-2:
/export 172.16.74.0/24

[root@vmw-test-1 ~]# mount -t nfs -overs=4 vmw-test-2:/ /mnt
[root@vmw-test-1 ~]# ls /mnt/export/
trondmy
[root@vmw-test-1 ~]# fuser -m /mnt
[root@vmw-test-1 ~]# umount /mnt
umount.nfs4: /mnt: device is busy
[root@vmw-test-1 ~]# fuser -m /mnt
[root@vmw-test-1 ~]# umount /mnt

I've attached the mountinfo output. It was created as follows:

[root@vmw-test-1 ~]# ls /mnt/export/
trondmy
[root@vmw-test-1 ~]# grep . `md5sum /proc/*/mountinfo|sort|uniq
-w32|awk '{print $2}'`>/tmp/mountdigest.txt
[root@vmw-test-1 ~]# umount /mnt
umount.nfs4: /mnt: device is busy

(so the mount was still busy).

Note that on my system, if I call 'umount' a second time after getting
the 'device is busy' error, then it succeeds. It looks as if the first
call to 'umount /mnt' causes the directory /mnt/export to clear,
causing the second 'umount /mnt' to succeeed (unless I try to access
/mnt/export again):

[root@vmw-test-1 ~]# mount -t nfs -overs=4 vmw-test-2:/ /mnt
[root@vmw-test-1 ~]# ls /mnt/export/
trondmy
[root@vmw-test-1 ~]# df
Filesystem              1K-blocks     Used Available Use% Mounted on
/dev/mapper/fedora-root  35740620 18968904  16771716  54% /
devtmpfs                  1947224        0   1947224   0% /dev
tmpfs                     1955120      156   1954964   1% /dev/shm
tmpfs                     1955120     1004   1954116   1% /run
tmpfs                     1955120        0   1955120   0% /sys/fs/cgroup
tmpfs                     1955120     1888   1953232   1% /tmp
/dev/sda1                 1998672    88396   1789036   5% /boot
vmw-test-2:/             35789824  8589824  27200000  25% /mnt
vmw-test-2:/export       35789824  8589824  27200000  25% /mnt/export
[root@vmw-test-1 ~]# umount /mnt
umount.nfs4: /mnt: device is busy
[root@vmw-test-1 ~]# df
Filesystem              1K-blocks     Used Available Use% Mounted on
/dev/mapper/fedora-root  35740620 18968904  16771716  54% /
devtmpfs                  1947224        0   1947224   0% /dev
tmpfs                     1955120      156   1954964   1% /dev/shm
tmpfs                     1955120     1004   1954116   1% /run
tmpfs                     1955120        0   1955120   0% /sys/fs/cgroup
tmpfs                     1955120     1892   1953228   1% /tmp
/dev/sda1                 1998672    88396   1789036   5% /boot
vmw-test-2:/             35789824  8590336  27199488  25% /mnt
[root@vmw-test-1 ~]# umount /mnt


-- 
Trond Myklebust

Linux NFS client maintainer, PrimaryData

trond.myklebust@primarydata.com

[-- Attachment #2: mountdigest.txt --]
[-- Type: text/plain, Size: 29534 bytes --]

/proc/1727/mountinfo:146 145 253:0 / / rw,relatime shared:127 master:1 - xfs /dev/mapper/fedora-root rw,seclabel,attr2,inode64,noquota
/proc/1727/mountinfo:147 146 0:5 / /dev rw,nosuid shared:128 master:2 - devtmpfs devtmpfs rw,seclabel,size=1947224k,nr_inodes=486806,mode=755
/proc/1727/mountinfo:148 147 0:17 / /dev/shm rw,nosuid,nodev shared:129 master:3 - tmpfs tmpfs rw,seclabel
/proc/1727/mountinfo:149 147 0:11 / /dev/pts rw,nosuid,noexec,relatime shared:130 master:4 - devpts devpts rw,seclabel,gid=5,mode=620,ptmxmode=000
/proc/1727/mountinfo:150 147 0:34 / /dev/hugepages rw,relatime shared:131 master:24 - hugetlbfs hugetlbfs rw,seclabel
/proc/1727/mountinfo:151 147 0:13 / /dev/mqueue rw,relatime shared:132 master:26 - mqueue mqueue rw,seclabel
/proc/1727/mountinfo:152 146 0:3 / /proc rw,nosuid,nodev,noexec,relatime shared:133 master:5 - proc proc rw
/proc/1727/mountinfo:153 152 0:33 / /proc/sys/fs/binfmt_misc rw,relatime shared:134 master:23 - autofs systemd-1 rw,fd=34,pgrp=1,timeout=300,minproto=5,maxproto=5,direct
/proc/1727/mountinfo:154 152 0:37 / /proc/fs/nfsd rw,relatime shared:135 master:29 - nfsd sunrpc rw
/proc/1727/mountinfo:155 146 0:15 / /sys rw,nosuid,nodev,noexec,relatime shared:136 master:6 - sysfs sysfs rw,seclabel
/proc/1727/mountinfo:156 155 0:16 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:137 master:7 - securityfs securityfs rw
/proc/1727/mountinfo:157 155 0:19 / /sys/fs/cgroup rw,nosuid,nodev,noexec shared:138 master:8 - tmpfs tmpfs rw,seclabel,mode=755
/proc/1727/mountinfo:158 157 0:20 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:139 master:9 - cgroup cgroup rw,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd
/proc/1727/mountinfo:159 157 0:22 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:140 master:10 - cgroup cgroup rw,cpuset
/proc/1727/mountinfo:160 157 0:23 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime shared:141 master:11 - cgroup cgroup rw,cpu,cpuacct
/proc/1727/mountinfo:161 157 0:24 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:142 master:12 - cgroup cgroup rw,memory
/proc/1727/mountinfo:162 157 0:25 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime shared:143 master:13 - cgroup cgroup rw,devices
/proc/1727/mountinfo:163 157 0:26 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime shared:144 master:14 - cgroup cgroup rw,freezer
/proc/1727/mountinfo:164 157 0:27 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime shared:145 master:15 - cgroup cgroup rw,net_cls,net_prio
/proc/1727/mountinfo:165 157 0:28 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime shared:146 master:16 - cgroup cgroup rw,blkio
/proc/1727/mountinfo:166 157 0:29 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime shared:147 master:17 - cgroup cgroup rw,perf_event
/proc/1727/mountinfo:167 157 0:30 / /sys/fs/cgroup/hugetlb rw,nosuid,nodev,noexec,relatime shared:148 master:18 - cgroup cgroup rw,hugetlb
/proc/1727/mountinfo:168 155 0:21 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:149 master:19 - pstore pstore rw
/proc/1727/mountinfo:169 155 0:31 / /sys/kernel/config rw,relatime shared:150 master:20 - configfs configfs rw
/proc/1727/mountinfo:170 155 0:14 / /sys/fs/selinux rw,relatime shared:151 master:21 - selinuxfs selinuxfs rw
/proc/1727/mountinfo:171 155 0:6 / /sys/kernel/debug rw,relatime shared:152 master:25 - debugfs debugfs rw
/proc/1727/mountinfo:172 146 0:18 / /run rw,nosuid,nodev shared:153 master:22 - tmpfs tmpfs rw,seclabel,mode=755
/proc/1727/mountinfo:173 146 0:35 / /tmp rw shared:154 master:27 - tmpfs tmpfs rw,seclabel
/proc/1727/mountinfo:174 146 0:36 / /var/lib/nfs/rpc_pipefs rw,relatime shared:156 master:28 - rpc_pipefs sunrpc rw
/proc/1727/mountinfo:175 146 8:1 / /boot rw,relatime shared:157 master:30 - ext4 /dev/sda1 rw,seclabel,data=ordered
/proc/1727/mountinfo:176 173 0:35 /systemd-named.service-XNco0iT/tmp /tmp rw shared:155 master:27 - tmpfs tmpfs rw,seclabel
/proc/1727/mountinfo:177 146 253:0 /var/tmp/systemd-named.service-XlTZT4V/tmp /var/tmp rw,relatime shared:158 master:1 - xfs /dev/mapper/fedora-root rw,seclabel,attr2,inode64,noquota
/proc/1727/mountinfo:213 155 0:38 / /sys/fs/fuse/connections rw,relatime shared:193 master:191 - fusectl fusectl rw
/proc/1727/mountinfo:219 172 0:39 / /run/user/52069/gvfs rw,nosuid,nodev,relatime shared:199 master:197 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=52069,group_id=52069
/proc/1727/mountinfo:261 146 0:40 / /mnt rw,relatime shared:240 master:237 - nfs4 vmw-test-2:/ rw,vers=4.0,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5p,clientaddr=172.16.74.146,local_lock=none,addr=172.16.74.147
/proc/1727/mountinfo:269 261 0:41 / /mnt/export rw,relatime shared:248 master:244 - nfs4 vmw-test-2:/export rw,vers=4.0,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=krb5p,clientaddr=172.16.74.146,local_lock=none,addr=172.16.74.147
/proc/1234/mountinfo:113 112 253:0 / / rw,relatime shared:95 master:1 - xfs /dev/mapper/fedora-root rw,seclabel,attr2,inode64,noquota
/proc/1234/mountinfo:114 113 0:5 / /dev rw,nosuid shared:96 master:2 - devtmpfs devtmpfs rw,seclabel,size=1947224k,nr_inodes=486806,mode=755
/proc/1234/mountinfo:115 114 0:17 / /dev/shm rw,nosuid,nodev shared:97 master:3 - tmpfs tmpfs rw,seclabel
/proc/1234/mountinfo:116 114 0:11 / /dev/pts rw,nosuid,noexec,relatime shared:98 master:4 - devpts devpts rw,seclabel,gid=5,mode=620,ptmxmode=000
/proc/1234/mountinfo:117 114 0:34 / /dev/hugepages rw,relatime shared:99 master:24 - hugetlbfs hugetlbfs rw,seclabel
/proc/1234/mountinfo:118 114 0:13 / /dev/mqueue rw,relatime shared:100 master:26 - mqueue mqueue rw,seclabel
/proc/1234/mountinfo:119 113 0:3 / /proc rw,nosuid,nodev,noexec,relatime shared:101 master:5 - proc proc rw
/proc/1234/mountinfo:120 119 0:33 / /proc/sys/fs/binfmt_misc rw,relatime shared:102 master:23 - autofs systemd-1 rw,fd=34,pgrp=1,timeout=300,minproto=5,maxproto=5,direct
/proc/1234/mountinfo:121 119 0:37 / /proc/fs/nfsd rw,relatime shared:103 master:29 - nfsd sunrpc rw
/proc/1234/mountinfo:122 113 0:15 / /sys rw,nosuid,nodev,noexec,relatime shared:104 master:6 - sysfs sysfs rw,seclabel
/proc/1234/mountinfo:123 122 0:16 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:105 master:7 - securityfs securityfs rw
/proc/1234/mountinfo:124 122 0:19 / /sys/fs/cgroup rw,nosuid,nodev,noexec shared:106 master:8 - tmpfs tmpfs rw,seclabel,mode=755
/proc/1234/mountinfo:125 124 0:20 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:107 master:9 - cgroup cgroup rw,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd
/proc/1234/mountinfo:126 124 0:22 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:108 master:10 - cgroup cgroup rw,cpuset
/proc/1234/mountinfo:127 124 0:23 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime shared:109 master:11 - cgroup cgroup rw,cpu,cpuacct
/proc/1234/mountinfo:128 124 0:24 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:110 master:12 - cgroup cgroup rw,memory
/proc/1234/mountinfo:129 124 0:25 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime shared:111 master:13 - cgroup cgroup rw,devices
/proc/1234/mountinfo:130 124 0:26 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime shared:112 master:14 - cgroup cgroup rw,freezer
/proc/1234/mountinfo:131 124 0:27 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime shared:113 master:15 - cgroup cgroup rw,net_cls,net_prio
/proc/1234/mountinfo:132 124 0:28 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime shared:114 master:16 - cgroup cgroup rw,blkio
/proc/1234/mountinfo:133 124 0:29 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime shared:115 master:17 - cgroup cgroup rw,perf_event
/proc/1234/mountinfo:134 124 0:30 / /sys/fs/cgroup/hugetlb rw,nosuid,nodev,noexec,relatime shared:116 master:18 - cgroup cgroup rw,hugetlb
/proc/1234/mountinfo:135 122 0:21 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:117 master:19 - pstore pstore rw
/proc/1234/mountinfo:136 122 0:31 / /sys/kernel/config rw,relatime shared:118 master:20 - configfs configfs rw
/proc/1234/mountinfo:137 122 0:14 / /sys/fs/selinux rw,relatime shared:119 master:21 - selinuxfs selinuxfs rw
/proc/1234/mountinfo:138 122 0:6 / /sys/kernel/debug rw,relatime shared:120 master:25 - debugfs debugfs rw
/proc/1234/mountinfo:139 113 0:18 / /run rw,nosuid,nodev shared:121 master:22 - tmpfs tmpfs rw,seclabel,mode=755
/proc/1234/mountinfo:140 113 0:35 / /tmp rw shared:122 master:27 - tmpfs tmpfs rw,seclabel
/proc/1234/mountinfo:141 113 0:36 / /var/lib/nfs/rpc_pipefs rw,relatime shared:124 master:28 - rpc_pipefs sunrpc rw
/proc/1234/mountinfo:142 113 8:1 / /boot rw,relatime shared:125 master:30 - ext4 /dev/sda1 rw,seclabel,data=ordered
/proc/1234/mountinfo:143 140 0:35 /systemd-colord.service-XwK2g4X/tmp /tmp rw shared:123 master:27 - tmpfs tmpfs rw,seclabel
/proc/1234/mountinfo:144 113 253:0 /var/tmp/systemd-colord.service-Xz7Z7wQ/tmp /var/tmp rw,relatime shared:126 master:1 - xfs /dev/mapper/fedora-root rw,seclabel,attr2,inode64,noquota
/proc/1234/mountinfo:214 122 0:38 / /sys/fs/fuse/connections rw,relatime shared:194 master:191 - fusectl fusectl rw
/proc/1234/mountinfo:220 139 0:39 / /run/user/52069/gvfs rw,nosuid,nodev,relatime shared:200 master:197 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=52069,group_id=52069
/proc/1234/mountinfo:262 113 0:40 / /mnt rw,relatime shared:241 master:237 - nfs4 vmw-test-2:/ rw,vers=4.0,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5p,clientaddr=172.16.74.146,local_lock=none,addr=172.16.74.147
/proc/1234/mountinfo:268 262 0:41 / /mnt/export rw,relatime shared:247 master:244 - nfs4 vmw-test-2:/export rw,vers=4.0,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=krb5p,clientaddr=172.16.74.146,local_lock=none,addr=172.16.74.147
/proc/587/mountinfo:53 47 0:3 / / rw,nosuid,nodev,noexec,relatime shared:37 master:5 - proc proc rw
/proc/587/mountinfo:54 53 0:33 / /sys/fs/binfmt_misc rw,relatime shared:38 master:23 - autofs systemd-1 rw,fd=34,pgrp=1,timeout=300,minproto=5,maxproto=5,direct
/proc/587/mountinfo:55 53 0:37 / /fs/nfsd rw,relatime shared:39 master:29 - nfsd sunrpc rw
/proc/1034/mountinfo:15 36 0:15 / /sys rw,nosuid,nodev,noexec,relatime shared:6 - sysfs sysfs rw,seclabel
/proc/1034/mountinfo:16 36 0:3 / /proc rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw
/proc/1034/mountinfo:17 36 0:5 / /dev rw,nosuid shared:2 - devtmpfs devtmpfs rw,seclabel,size=1947224k,nr_inodes=486806,mode=755
/proc/1034/mountinfo:18 15 0:16 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:7 - securityfs securityfs rw
/proc/1034/mountinfo:19 17 0:17 / /dev/shm rw,nosuid,nodev shared:3 - tmpfs tmpfs rw,seclabel
/proc/1034/mountinfo:20 17 0:11 / /dev/pts rw,nosuid,noexec,relatime shared:4 - devpts devpts rw,seclabel,gid=5,mode=620,ptmxmode=000
/proc/1034/mountinfo:21 36 0:18 / /run rw,nosuid,nodev shared:22 - tmpfs tmpfs rw,seclabel,mode=755
/proc/1034/mountinfo:22 15 0:19 / /sys/fs/cgroup rw,nosuid,nodev,noexec shared:8 - tmpfs tmpfs rw,seclabel,mode=755
/proc/1034/mountinfo:23 22 0:20 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:9 - cgroup cgroup rw,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd
/proc/1034/mountinfo:24 15 0:21 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:19 - pstore pstore rw
/proc/1034/mountinfo:25 22 0:22 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:10 - cgroup cgroup rw,cpuset
/proc/1034/mountinfo:26 22 0:23 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime shared:11 - cgroup cgroup rw,cpu,cpuacct
/proc/1034/mountinfo:27 22 0:24 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:12 - cgroup cgroup rw,memory
/proc/1034/mountinfo:28 22 0:25 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime shared:13 - cgroup cgroup rw,devices
/proc/1034/mountinfo:29 22 0:26 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime shared:14 - cgroup cgroup rw,freezer
/proc/1034/mountinfo:30 22 0:27 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime shared:15 - cgroup cgroup rw,net_cls,net_prio
/proc/1034/mountinfo:31 22 0:28 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime shared:16 - cgroup cgroup rw,blkio
/proc/1034/mountinfo:32 22 0:29 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime shared:17 - cgroup cgroup rw,perf_event
/proc/1034/mountinfo:33 22 0:30 / /sys/fs/cgroup/hugetlb rw,nosuid,nodev,noexec,relatime shared:18 - cgroup cgroup rw,hugetlb
/proc/1034/mountinfo:34 15 0:31 / /sys/kernel/config rw,relatime shared:20 - configfs configfs rw
/proc/1034/mountinfo:36 0 253:0 / / rw,relatime shared:1 - xfs /dev/mapper/fedora-root rw,seclabel,attr2,inode64,noquota
/proc/1034/mountinfo:37 15 0:14 / /sys/fs/selinux rw,relatime shared:21 - selinuxfs selinuxfs rw
/proc/1034/mountinfo:38 16 0:33 / /proc/sys/fs/binfmt_misc rw,relatime shared:23 - autofs systemd-1 rw,fd=34,pgrp=1,timeout=300,minproto=5,maxproto=5,direct
/proc/1034/mountinfo:39 17 0:34 / /dev/hugepages rw,relatime shared:24 - hugetlbfs hugetlbfs rw,seclabel
/proc/1034/mountinfo:40 15 0:6 / /sys/kernel/debug rw,relatime shared:25 - debugfs debugfs rw
/proc/1034/mountinfo:41 17 0:13 / /dev/mqueue rw,relatime shared:26 - mqueue mqueue rw,seclabel
/proc/1034/mountinfo:42 36 0:35 / /tmp rw shared:27 - tmpfs tmpfs rw,seclabel
/proc/1034/mountinfo:43 36 0:36 / /var/lib/nfs/rpc_pipefs rw,relatime shared:28 - rpc_pipefs sunrpc rw
/proc/1034/mountinfo:44 16 0:37 / /proc/fs/nfsd rw,relatime shared:29 - nfsd sunrpc rw
/proc/1034/mountinfo:45 36 8:1 / /boot rw,relatime shared:30 - ext4 /dev/sda1 rw,seclabel,data=ordered
/proc/1034/mountinfo:211 15 0:38 / /sys/fs/fuse/connections rw,relatime shared:191 - fusectl fusectl rw
/proc/1034/mountinfo:217 21 0:39 / /run/user/52069/gvfs rw,nosuid,nodev,relatime shared:197 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=52069,group_id=52069
/proc/1034/mountinfo:258 36 0:40 / /mnt rw,relatime shared:237 - nfs4 vmw-test-2:/ rw,vers=4.0,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5p,clientaddr=172.16.74.146,local_lock=none,addr=172.16.74.147
/proc/1034/mountinfo:265 258 0:41 / /mnt/export rw,relatime shared:244 - nfs4 vmw-test-2:/export rw,vers=4.0,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=krb5p,clientaddr=172.16.74.146,local_lock=none,addr=172.16.74.147
/proc/23/mountinfo:5 4 0:5 / / rw,relatime - devtmpfs devtmpfs rw,seclabel,size=1947224k,nr_inodes=486806,mode=755
/proc/638/mountinfo:80 79 253:0 / / rw,relatime shared:63 master:1 - xfs /dev/mapper/fedora-root rw,seclabel,attr2,inode64,noquota
/proc/638/mountinfo:81 80 0:5 / /dev rw,nosuid shared:64 master:2 - devtmpfs devtmpfs rw,seclabel,size=1947224k,nr_inodes=486806,mode=755
/proc/638/mountinfo:82 81 0:17 / /dev/shm rw,nosuid,nodev shared:65 master:3 - tmpfs tmpfs rw,seclabel
/proc/638/mountinfo:83 81 0:11 / /dev/pts rw,nosuid,noexec,relatime shared:66 master:4 - devpts devpts rw,seclabel,gid=5,mode=620,ptmxmode=000
/proc/638/mountinfo:84 81 0:34 / /dev/hugepages rw,relatime shared:67 master:24 - hugetlbfs hugetlbfs rw,seclabel
/proc/638/mountinfo:85 81 0:13 / /dev/mqueue rw,relatime shared:68 master:26 - mqueue mqueue rw,seclabel
/proc/638/mountinfo:86 80 0:3 / /proc rw,nosuid,nodev,noexec,relatime shared:69 master:5 - proc proc rw
/proc/638/mountinfo:87 86 0:33 / /proc/sys/fs/binfmt_misc rw,relatime shared:70 master:23 - autofs systemd-1 rw,fd=34,pgrp=1,timeout=300,minproto=5,maxproto=5,direct
/proc/638/mountinfo:88 86 0:37 / /proc/fs/nfsd rw,relatime shared:71 master:29 - nfsd sunrpc rw
/proc/638/mountinfo:89 80 0:15 / /sys rw,nosuid,nodev,noexec,relatime shared:72 master:6 - sysfs sysfs rw,seclabel
/proc/638/mountinfo:90 89 0:16 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:73 master:7 - securityfs securityfs rw
/proc/638/mountinfo:91 89 0:19 / /sys/fs/cgroup rw,nosuid,nodev,noexec shared:74 master:8 - tmpfs tmpfs rw,seclabel,mode=755
/proc/638/mountinfo:92 91 0:20 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:75 master:9 - cgroup cgroup rw,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd
/proc/638/mountinfo:93 91 0:22 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:76 master:10 - cgroup cgroup rw,cpuset
/proc/638/mountinfo:94 91 0:23 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime shared:77 master:11 - cgroup cgroup rw,cpu,cpuacct
/proc/638/mountinfo:95 91 0:24 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:78 master:12 - cgroup cgroup rw,memory
/proc/638/mountinfo:96 91 0:25 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime shared:79 master:13 - cgroup cgroup rw,devices
/proc/638/mountinfo:97 91 0:26 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime shared:80 master:14 - cgroup cgroup rw,freezer
/proc/638/mountinfo:98 91 0:27 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime shared:81 master:15 - cgroup cgroup rw,net_cls,net_prio
/proc/638/mountinfo:99 91 0:28 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime shared:82 master:16 - cgroup cgroup rw,blkio
/proc/638/mountinfo:100 91 0:29 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime shared:83 master:17 - cgroup cgroup rw,perf_event
/proc/638/mountinfo:101 91 0:30 / /sys/fs/cgroup/hugetlb rw,nosuid,nodev,noexec,relatime shared:84 master:18 - cgroup cgroup rw,hugetlb
/proc/638/mountinfo:102 89 0:21 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:85 master:19 - pstore pstore rw
/proc/638/mountinfo:103 89 0:31 / /sys/kernel/config rw,relatime shared:86 master:20 - configfs configfs rw
/proc/638/mountinfo:104 89 0:14 / /sys/fs/selinux rw,relatime shared:87 master:21 - selinuxfs selinuxfs rw
/proc/638/mountinfo:105 89 0:6 / /sys/kernel/debug rw,relatime shared:88 master:25 - debugfs debugfs rw
/proc/638/mountinfo:106 80 0:18 / /run rw,nosuid,nodev shared:89 master:22 - tmpfs tmpfs rw,seclabel,mode=755
/proc/638/mountinfo:107 80 0:35 / /tmp rw shared:90 master:27 - tmpfs tmpfs rw,seclabel
/proc/638/mountinfo:108 80 0:36 / /var/lib/nfs/rpc_pipefs rw,relatime shared:92 master:28 - rpc_pipefs sunrpc rw
/proc/638/mountinfo:109 80 8:1 / /boot rw,relatime shared:93 master:30 - ext4 /dev/sda1 rw,seclabel,data=ordered
/proc/638/mountinfo:110 107 0:35 /systemd-ntpd.service-XeEvQUb/tmp /tmp rw shared:91 master:27 - tmpfs tmpfs rw,seclabel
/proc/638/mountinfo:111 80 253:0 /var/tmp/systemd-ntpd.service-XpK3O4X/tmp /var/tmp rw,relatime shared:94 master:1 - xfs /dev/mapper/fedora-root rw,seclabel,attr2,inode64,noquota
/proc/638/mountinfo:215 89 0:38 / /sys/fs/fuse/connections rw,relatime shared:195 master:191 - fusectl fusectl rw
/proc/638/mountinfo:221 106 0:39 / /run/user/52069/gvfs rw,nosuid,nodev,relatime shared:201 master:197 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=52069,group_id=52069
/proc/638/mountinfo:263 80 0:40 / /mnt rw,relatime shared:242 master:237 - nfs4 vmw-test-2:/ rw,vers=4.0,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5p,clientaddr=172.16.74.146,local_lock=none,addr=172.16.74.147
/proc/638/mountinfo:267 263 0:41 / /mnt/export rw,relatime shared:246 master:244 - nfs4 vmw-test-2:/export rw,vers=4.0,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=krb5p,clientaddr=172.16.74.146,local_lock=none,addr=172.16.74.147
/proc/2245/mountinfo:224 223 253:0 / / rw,relatime shared:203 master:1 - xfs /dev/mapper/fedora-root rw,seclabel,attr2,inode64,noquota
/proc/2245/mountinfo:225 224 0:5 / /dev rw,nosuid shared:204 master:2 - devtmpfs devtmpfs rw,seclabel,size=1947224k,nr_inodes=486806,mode=755
/proc/2245/mountinfo:226 225 0:17 / /dev/shm rw,nosuid,nodev shared:205 master:3 - tmpfs tmpfs rw,seclabel
/proc/2245/mountinfo:227 225 0:11 / /dev/pts rw,nosuid,noexec,relatime shared:206 master:4 - devpts devpts rw,seclabel,gid=5,mode=620,ptmxmode=000
/proc/2245/mountinfo:228 225 0:34 / /dev/hugepages rw,relatime shared:207 master:24 - hugetlbfs hugetlbfs rw,seclabel
/proc/2245/mountinfo:229 225 0:13 / /dev/mqueue rw,relatime shared:208 master:26 - mqueue mqueue rw,seclabel
/proc/2245/mountinfo:230 224 0:3 / /proc rw,nosuid,nodev,noexec,relatime shared:209 master:5 - proc proc rw
/proc/2245/mountinfo:231 230 0:33 / /proc/sys/fs/binfmt_misc rw,relatime shared:210 master:23 - autofs systemd-1 rw,fd=34,pgrp=1,timeout=300,minproto=5,maxproto=5,direct
/proc/2245/mountinfo:232 230 0:37 / /proc/fs/nfsd rw,relatime shared:211 master:29 - nfsd sunrpc rw
/proc/2245/mountinfo:233 224 0:15 / /sys rw,nosuid,nodev,noexec,relatime shared:212 master:6 - sysfs sysfs rw,seclabel
/proc/2245/mountinfo:234 233 0:16 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:213 master:7 - securityfs securityfs rw
/proc/2245/mountinfo:235 233 0:19 / /sys/fs/cgroup rw,nosuid,nodev,noexec shared:214 master:8 - tmpfs tmpfs rw,seclabel,mode=755
/proc/2245/mountinfo:236 235 0:20 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:215 master:9 - cgroup cgroup rw,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd
/proc/2245/mountinfo:237 235 0:22 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:216 master:10 - cgroup cgroup rw,cpuset
/proc/2245/mountinfo:238 235 0:23 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime shared:217 master:11 - cgroup cgroup rw,cpu,cpuacct
/proc/2245/mountinfo:239 235 0:24 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:218 master:12 - cgroup cgroup rw,memory
/proc/2245/mountinfo:240 235 0:25 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime shared:219 master:13 - cgroup cgroup rw,devices
/proc/2245/mountinfo:241 235 0:26 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime shared:220 master:14 - cgroup cgroup rw,freezer
/proc/2245/mountinfo:242 235 0:27 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime shared:221 master:15 - cgroup cgroup rw,net_cls,net_prio
/proc/2245/mountinfo:243 235 0:28 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime shared:222 master:16 - cgroup cgroup rw,blkio
/proc/2245/mountinfo:244 235 0:29 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime shared:223 master:17 - cgroup cgroup rw,perf_event
/proc/2245/mountinfo:245 235 0:30 / /sys/fs/cgroup/hugetlb rw,nosuid,nodev,noexec,relatime shared:224 master:18 - cgroup cgroup rw,hugetlb
/proc/2245/mountinfo:246 233 0:21 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:225 master:19 - pstore pstore rw
/proc/2245/mountinfo:247 233 0:31 / /sys/kernel/config rw,relatime shared:226 master:20 - configfs configfs rw
/proc/2245/mountinfo:248 233 0:14 / /sys/fs/selinux rw,relatime shared:227 master:21 - selinuxfs selinuxfs rw
/proc/2245/mountinfo:249 233 0:6 / /sys/kernel/debug rw,relatime shared:228 master:25 - debugfs debugfs rw
/proc/2245/mountinfo:250 233 0:38 / /sys/fs/fuse/connections rw,relatime shared:229 master:191 - fusectl fusectl rw
/proc/2245/mountinfo:251 224 0:18 / /run rw,nosuid,nodev shared:230 master:22 - tmpfs tmpfs rw,seclabel,mode=755
/proc/2245/mountinfo:252 251 0:39 / /run/user/52069/gvfs rw,nosuid,nodev,relatime shared:231 master:197 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=52069,group_id=52069
/proc/2245/mountinfo:253 224 0:35 / /tmp rw shared:232 master:27 - tmpfs tmpfs rw,seclabel
/proc/2245/mountinfo:254 224 0:36 / /var/lib/nfs/rpc_pipefs rw,relatime shared:234 master:28 - rpc_pipefs sunrpc rw
/proc/2245/mountinfo:255 224 8:1 / /boot rw,relatime shared:235 master:30 - ext4 /dev/sda1 rw,seclabel,data=ordered
/proc/2245/mountinfo:256 253 0:35 /systemd-cups.service-X7j9dtS/tmp /tmp rw shared:233 master:27 - tmpfs tmpfs rw,seclabel
/proc/2245/mountinfo:257 224 253:0 /var/tmp/systemd-cups.service-X2k26ji/tmp /var/tmp rw,relatime shared:236 master:1 - xfs /dev/mapper/fedora-root rw,seclabel,attr2,inode64,noquota
/proc/2245/mountinfo:259 224 0:40 / /mnt rw,relatime shared:238 master:237 - nfs4 vmw-test-2:/ rw,vers=4.0,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5p,clientaddr=172.16.74.146,local_lock=none,addr=172.16.74.147
/proc/2245/mountinfo:271 259 0:41 / /mnt/export rw,relatime shared:250 master:244 - nfs4 vmw-test-2:/export rw,vers=4.0,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=krb5p,clientaddr=172.16.74.146,local_lock=none,addr=172.16.74.147
/proc/1753/mountinfo:179 178 253:0 / / rw,relatime shared:159 master:1 - xfs /dev/mapper/fedora-root rw,seclabel,attr2,inode64,noquota
/proc/1753/mountinfo:180 179 0:5 / /dev rw,nosuid shared:160 master:2 - devtmpfs devtmpfs rw,seclabel,size=1947224k,nr_inodes=486806,mode=755
/proc/1753/mountinfo:181 180 0:17 / /dev/shm rw,nosuid,nodev shared:161 master:3 - tmpfs tmpfs rw,seclabel
/proc/1753/mountinfo:182 180 0:11 / /dev/pts rw,nosuid,noexec,relatime shared:162 master:4 - devpts devpts rw,seclabel,gid=5,mode=620,ptmxmode=000
/proc/1753/mountinfo:183 180 0:34 / /dev/hugepages rw,relatime shared:163 master:24 - hugetlbfs hugetlbfs rw,seclabel
/proc/1753/mountinfo:184 180 0:13 / /dev/mqueue rw,relatime shared:164 master:26 - mqueue mqueue rw,seclabel
/proc/1753/mountinfo:185 179 0:3 / /proc rw,nosuid,nodev,noexec,relatime shared:165 master:5 - proc proc rw
/proc/1753/mountinfo:186 185 0:33 / /proc/sys/fs/binfmt_misc rw,relatime shared:166 master:23 - autofs systemd-1 rw,fd=34,pgrp=1,timeout=300,minproto=5,maxproto=5,direct
/proc/1753/mountinfo:187 185 0:37 / /proc/fs/nfsd rw,relatime shared:167 master:29 - nfsd sunrpc rw
/proc/1753/mountinfo:188 179 0:15 / /sys rw,nosuid,nodev,noexec,relatime shared:168 master:6 - sysfs sysfs rw,seclabel
/proc/1753/mountinfo:189 188 0:16 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:169 master:7 - securityfs securityfs rw
/proc/1753/mountinfo:190 188 0:19 / /sys/fs/cgroup rw,nosuid,nodev,noexec shared:170 master:8 - tmpfs tmpfs rw,seclabel,mode=755
/proc/1753/mountinfo:191 190 0:20 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:171 master:9 - cgroup cgroup rw,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd
/proc/1753/mountinfo:192 190 0:22 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:172 master:10 - cgroup cgroup rw,cpuset
/proc/1753/mountinfo:193 190 0:23 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime shared:173 master:11 - cgroup cgroup rw,cpu,cpuacct
/proc/1753/mountinfo:194 190 0:24 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:174 master:12 - cgroup cgroup rw,memory
/proc/1753/mountinfo:195 190 0:25 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime shared:175 master:13 - cgroup cgroup rw,devices
/proc/1753/mountinfo:196 190 0:26 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime shared:176 master:14 - cgroup cgroup rw,freezer
/proc/1753/mountinfo:197 190 0:27 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime shared:177 master:15 - cgroup cgroup rw,net_cls,net_prio
/proc/1753/mountinfo:198 190 0:28 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime shared:178 master:16 - cgroup cgroup rw,blkio
/proc/1753/mountinfo:199 190 0:29 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime shared:179 master:17 - cgroup cgroup rw,perf_event
/proc/1753/mountinfo:200 190 0:30 / /sys/fs/cgroup/hugetlb rw,nosuid,nodev,noexec,relatime shared:180 master:18 - cgroup cgroup rw,hugetlb
/proc/1753/mountinfo:201 188 0:21 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:181 master:19 - pstore pstore rw
/proc/1753/mountinfo:202 188 0:31 / /sys/kernel/config rw,relatime shared:182 master:20 - configfs configfs rw
/proc/1753/mountinfo:203 188 0:14 / /sys/fs/selinux rw,relatime shared:183 master:21 - selinuxfs selinuxfs rw
/proc/1753/mountinfo:204 188 0:6 / /sys/kernel/debug rw,relatime shared:184 master:25 - debugfs debugfs rw
/proc/1753/mountinfo:205 179 0:18 / /run rw,nosuid,nodev shared:185 master:22 - tmpfs tmpfs rw,seclabel,mode=755
/proc/1753/mountinfo:206 179 0:35 / /tmp rw shared:186 master:27 - tmpfs tmpfs rw,seclabel
/proc/1753/mountinfo:207 179 0:36 / /var/lib/nfs/rpc_pipefs rw,relatime shared:188 master:28 - rpc_pipefs sunrpc rw
/proc/1753/mountinfo:208 179 8:1 / /boot rw,relatime shared:189 master:30 - ext4 /dev/sda1 rw,seclabel,data=ordered
/proc/1753/mountinfo:209 206 0:35 /systemd-httpd.service-XuDh48b/tmp /tmp rw shared:187 master:27 - tmpfs tmpfs rw,seclabel
/proc/1753/mountinfo:210 179 253:0 /var/tmp/systemd-httpd.service-XzNRgds/tmp /var/tmp rw,relatime shared:190 master:1 - xfs /dev/mapper/fedora-root rw,seclabel,attr2,inode64,noquota
/proc/1753/mountinfo:212 188 0:38 / /sys/fs/fuse/connections rw,relatime shared:192 master:191 - fusectl fusectl rw
/proc/1753/mountinfo:218 205 0:39 / /run/user/52069/gvfs rw,nosuid,nodev,relatime shared:198 master:197 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=52069,group_id=52069
/proc/1753/mountinfo:260 179 0:40 / /mnt rw,relatime shared:239 master:237 - nfs4 vmw-test-2:/ rw,vers=4.0,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5p,clientaddr=172.16.74.146,local_lock=none,addr=172.16.74.147
/proc/1753/mountinfo:270 260 0:41 / /mnt/export rw,relatime shared:249 master:244 - nfs4 vmw-test-2:/export rw,vers=4.0,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=krb5p,clientaddr=172.16.74.146,local_lock=none,addr=172.16.74.147

Re: VFS regression: commit aba809cf0944 breaks MNT_SHRINKABLE automounted partitions

[Al Viro]

On Fri, Aug 29, 2014 at 05:47:58PM -0400, Trond Myklebust wrote:

> Note that the issue happens with NFSv4 or NFSv4.1 (not NFSv3).

That's what I'd been using for testing.

> Note that on my system, if I call 'umount' a second time after getting
> the 'device is busy' error, then it succeeds. It looks as if the first
> call to 'umount /mnt' causes the directory /mnt/export to clear,
> causing the second 'umount /mnt' to succeeed (unless I try to access
> /mnt/export again):

Now, that smells like a different bug - see if commit ab8f2c from -next
helps with that one.

Anyway, I think I see what's going on; moreover, I have a kinda-sorta
solution for the current tree, but it's not something that'll be fun
to backport and I'd prefer to look for alternative solutions before going
for that one.  Hopefully I'll have something postable by tomorrow morning...

Re: VFS regression: commit aba809cf0944 breaks MNT_SHRINKABLE automounted partitions

[Peng Tao]

On Sat, Aug 30, 2014 at 7:01 AM, Al Viro <viro@zeniv.linux.org.uk> wrote:
> On Fri, Aug 29, 2014 at 05:47:58PM -0400, Trond Myklebust wrote:
>
>> Note that the issue happens with NFSv4 or NFSv4.1 (not NFSv3).
>
> That's what I'd been using for testing.
>
>> Note that on my system, if I call 'umount' a second time after getting
>> the 'device is busy' error, then it succeeds. It looks as if the first
>> call to 'umount /mnt' causes the directory /mnt/export to clear,
>> causing the second 'umount /mnt' to succeeed (unless I try to access
>> /mnt/export again):
Just to be clear, it is what I saw as well. I did not get permanent
-EBUSY and I could umount /mnt if I run `umount /mnt` twice.

And I did not do git bisect. I had doubts in the code so I found the
specific commit to revert it and it worked. But if it is some other
commit in the middle, I could have missed it.

My theory to blame aba809cf0944 is that in
do_umount()->shrink_submounts()->umount_tree(), it transfers parent
mnt refcount (held by child mnt) to mnt_ex_mountpoint but that is put
_after_ do_umount() calls propagate_mount_busy(mnt, 2). So
propagate_mount_busy() would fail do_refcount_check(). Does it make
sense to you?

>
> Now, that smells like a different bug - see if commit ab8f2c from -next
> helps with that one.
>
I cherri-picked it and it still failed with -EBUSY.

Cheers,
Tao

Re: VFS regression: commit aba809cf0944 breaks MNT_SHRINKABLE automounted partitions

[Al Viro]

On Sat, Aug 30, 2014 at 08:38:56AM +0800, Peng Tao wrote:
> On Sat, Aug 30, 2014 at 7:01 AM, Al Viro <viro@zeniv.linux.org.uk> wrote:
> > On Fri, Aug 29, 2014 at 05:47:58PM -0400, Trond Myklebust wrote:
> >
> >> Note that the issue happens with NFSv4 or NFSv4.1 (not NFSv3).
> >
> > That's what I'd been using for testing.
> >
> >> Note that on my system, if I call 'umount' a second time after getting
> >> the 'device is busy' error, then it succeeds. It looks as if the first
> >> call to 'umount /mnt' causes the directory /mnt/export to clear,
> >> causing the second 'umount /mnt' to succeeed (unless I try to access
> >> /mnt/export again):
> Just to be clear, it is what I saw as well. I did not get permanent
> -EBUSY and I could umount /mnt if I run `umount /mnt` twice.
> 
> And I did not do git bisect. I had doubts in the code so I found the
> specific commit to revert it and it worked. But if it is some other
> commit in the middle, I could have missed it.
> 
> My theory to blame aba809cf0944 is that in
> do_umount()->shrink_submounts()->umount_tree(), it transfers parent
> mnt refcount (held by child mnt) to mnt_ex_mountpoint but that is put
> _after_ do_umount() calls propagate_mount_busy(mnt, 2). So
> propagate_mount_busy() would fail do_refcount_check(). Does it make
> sense to you?

Of course; the question is what to do with that.  The root of the problem
is in ordering constraints on fs shutdown and dput() of mountpoints.  What
we have right now solves that by holding "ghost" reference to parent vfsmount
until namespace_unlock(), where we drop them after having done that dput().

The constrants are
	* dput() of mountpoint should happen only after we'd dropped namespace
lock - we really don't want any IO under it.
	* it should happen before fs containing that mountpoint gets shut down.
	* non-lazy umount(2) should not return until all filesystems it
will shut down are, indeed, shut down

FWIW, I suspect that the simplest solution is this:
	* decrement parent's refcount when detaching a child from it in
umount_tree()
	* in namespace_unlock(), _before_ dropping the rwsem, go through the
list of victims and bump refcounts on their parents.  Note that those could
not have been shut down, let alone freed yet - either we are the ones to
take the parent out, in which case its refcount won't get to zero until
namespace_unlock() regardless of the children, or we are not.  In which case
its ->mnt_ns is non-NULL and will stay that way until rwsem is dropped (and
its refcount is also guaranteed to stay positive, not that mntput() would
bother checking that with non-NULL ->mnt_ns).

It does give a window when parent looks busy - from dropping rwsem in
namespace_unlock() to the moment when we get to the (ex-)child in the
loop there.  So what?  It doesn't affect the busy checks for that syscall -
we'd already done them all.  It doesn't affect the busy checks for anything
we do after returning from that syscall - namespace_unlock() completes before
we return to userland and all is good.  And if another process calls umount()
before ours return...  Well, it gets its EBUSY - same as it would if it got
there just as the first one has finished finding the mountpoint of child.
Two processes playing with umount(2) on the same area in the tree without
serialization should be ready to see the damn things transiently busy, simply
from another process looking at its victims...

IOW, how about this (completely untested)?  That's on top of that commit
from -next, but those two should be easy to backport to older trees
(->mnt_mp_list won't be there for those, but that's about it).

diff --git a/fs/namespace.c b/fs/namespace.c
index 3733cfb..00e5b1e 100644
--- a/fs/namespace.c
+++ b/fs/namespace.c
@@ -1299,6 +1299,11 @@ static void namespace_unlock(void)
 	head.first->pprev = &head.first;
 	INIT_HLIST_HEAD(&unmounted);
 
+	/* undo decrements we'd done in umount_tree() */
+	hlist_for_each_entry(mnt, &head, mnt_hash)
+		if (mnt->mnt_ex_mountpoint.mnt)
+			mntget(mnt->mnt_ex_mountpoint.mnt);
+
 	up_write(&namespace_sem);
 
 	synchronize_rcu();
@@ -1351,6 +1356,7 @@ void umount_tree(struct mount *mnt, int how)
 		if (mnt_has_parent(p)) {
 			hlist_del_init(&p->mnt_mp_list);
 			put_mountpoint(p->mnt_mp);
+			mnt_add_count(p->mnt_parent, -1);
 			/* move the reference to mountpoint into ->mnt_ex_mountpoint */
 			p->mnt_ex_mountpoint.dentry = p->mnt_mountpoint;
 			p->mnt_ex_mountpoint.mnt = &p->mnt_parent->mnt;

Re: VFS regression: commit aba809cf0944 breaks MNT_SHRINKABLE automounted partitions

[Al Viro]

	Could you check if vfs.git#for-linus fixes that crap?

Re: VFS regression: commit aba809cf0944 breaks MNT_SHRINKABLE automounted partitions

[Peng Tao]

On Sun, Aug 31, 2014 at 7:27 AM, Al Viro <viro@zeniv.linux.org.uk> wrote:
>         Could you check if vfs.git#for-linus fixes that crap?
The related commits are the top two of vfs.git#for-linus, right? I
cherry-picked them to 3.14.15 and they fixed the bug instantly. Thank
you for the quick fix!

Cheers,
Tao

Re: VFS regression: commit aba809cf0944 breaks MNT_SHRINKABLE automounted partitions

[Al Viro]

On Sun, Aug 31, 2014 at 08:31:47AM +0800, Peng Tao wrote:
> On Sun, Aug 31, 2014 at 7:27 AM, Al Viro <viro@zeniv.linux.org.uk> wrote:
> >         Could you check if vfs.git#for-linus fixes that crap?
> The related commits are the top two of vfs.git#for-linus, right? I
> cherry-picked them to 3.14.15 and they fixed the bug instantly. Thank
> you for the quick fix!

Folks, could somebody put together an xfstests (or LTP, for that matter)
patch adding that testcase?  Which regression suite is normally used
for NFS work, BTW?  It's not, strictly speaking, an NFS-specific bug
(you can trigger the same thing on AFS and CIFS as well), but NFS is
by far the most common of those three, so...

Re: VFS regression: commit aba809cf0944 breaks MNT_SHRINKABLE automounted partitions

[Dave Chinner]

On Sun, Aug 31, 2014 at 05:09:10AM +0100, Al Viro wrote:
> On Sun, Aug 31, 2014 at 08:31:47AM +0800, Peng Tao wrote:
> > On Sun, Aug 31, 2014 at 7:27 AM, Al Viro <viro@zeniv.linux.org.uk> wrote:
> > >         Could you check if vfs.git#for-linus fixes that crap?
> > The related commits are the top two of vfs.git#for-linus, right? I
> > cherry-picked them to 3.14.15 and they fixed the bug instantly. Thank
> > you for the quick fix!
> 
> Folks, could somebody put together an xfstests (or LTP, for that matter)
> patch adding that testcase?  Which regression suite is normally used
> for NFS work, BTW?  It's not, strictly speaking, an NFS-specific bug
> (you can trigger the same thing on AFS and CIFS as well), but NFS is
> by far the most common of those three, so...

You can use xfstests to test NFS from the client side, and there are
patches pending to add CIFS client support to xfstests as well.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com