Re: why is the same mount point repeatedly mounted in nested manner?

[Goffredo Baroncelli <kreijack@inwind.it>]

On 8/10/21 6:27 PM, Dave T wrote:
> On Tue, Aug 10, 2021 at 12:17 PM Goffredo Baroncelli <kreijack@inwind.it> wrote:
>>
>> On 8/10/21 6:03 PM, Dave T wrote:
>>> On Tue, Aug 10, 2021 at 11:43 AM Goffredo Baroncelli <kreijack@inwind.it> wrote:
[...]
>> What is more interesting is the dependencies between srv-nfs-var-cache-pacman.mount and nfs-server.service. I suspect (but I don't have any proof) that systemd is confused by the tuple {btrfs subvolume, bind mount, nfs dependecies}.
>>
>> What happens if you restart the nfs-server ?
> 
> As part of this issue, nfs clients have been experiencing slowness and
> sometimes hangs. I have restarted the nfs server service a few times
> while this issue was happening and it seems to temporarily resolve the
> client issues, but I'm not 100% sure because a specific incident will
> resolve eventually on its own without any intervention by me. But
> future incidences of slowness or temporary hangs are continuing and I
> do think this is all related.
> 
> Restarting the nfs server service does not make the nested mounts we
> are discussing go away. I have to unwind them with multiple calls to
> umount -- although the number of calls is less than the number of
> nested mounts, which I find confusing. Typically just 2-3 calls to
> umount will unmount a nesting 5-6 deep.
> 
> Clients connect to a number of different nfs shares from this server,
> yet none of the others wind up with nested mounts like
> srv-nfs-var-cache-pacman.mount . All mounts are configured the same
> way, using the same bind mount parameters in fstab.
> 

Unfortunately I don't have any more suggestions..

My opinion is that the problem is not related to the btrfs itself, but it is more a side effect of the interaction between systemd and btrfs. What about looking to the log ?

$ sudo journalctl -u  srv-nfs-var-cache-pacman.mount

Then check the log near the mount/unmount

To avoid this problem enterely, what about using a symlink instead of the bind mount ?

Something like

# rmdir /srv/nfs/var/cache/pacman ; ln -sf /var/cache/pacman /srv/nfs/var/cache/pacman


BR
GB

>>>
>>>>
>>>> [...]
>>>>>
>>>>> As mentioned, I have (temporarily) stopped unmounting these volumes
>>>>> and I just leave them mounted all the time. The logs now look like
>>>>> this:
>>>>>
>>>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: mounting btrbk btrtop volumes...
>>>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/root] (1 of
>>>>> 3) was already mounted. Nothing to do.
>>>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/home] (2 of
>>>>> 3) was already mounted. Nothing to do.
>>>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: INFO: [/mnt/btrtop/user] (3 of
>>>>> 3) was already mounted. Nothing to do.
>>>>
>>>> This told another story. It seems that "btrbk" itself already try to mount the btrfs subvolume. I understood that it was the systemd unit to do that. Could you share the content of btrbk_run.sh ?
>>>>
>>>
>>> # systemctl cat btrbk.service
>>> # /usr/lib/systemd/system/btrbk.service
>>> [Unit]
>>> Description=btrbk backup
>>> Documentation=man:btrbk(1)
>>>
>>> [Service]
>>> Type=oneshot
>>> ExecStart=/usr/bin/btrbk run
>>>
>>> # /etc/systemd/system/btrbk.service.d/override.conf
>>> [Service]
>>> ExecStart=
>>> ExecStart=/usr/local/bin/btrbk_run.sh
>>>
>>>
>>> # cat /usr/local/bin/btrbk_run.sh
>>> #!/bin/bash
>>>
>>> /usr/local/bin/btrbk_mount
>>>
>>> /usr/bin/btrbk --config /etc/btrbk/btrbk.conf run
>>>
>>> # 2021-08-05 My first troubleshooting step is to disable unmounting
>>> these shares.
>>> # /usr/local/bin/btrbk-umount
>>>
>>>
>>> # cat /usr/local/bin/btrbk_mount
>>> #!/bin/bash
>>>
>>> btrbk_mount() {
>>>
>>> echo "mounting btrbk btrtop volumes..."
>>>
>>> findmnt /mnt/btrtop/root
>>> if [[ $? -ne 0 ]]; then \
>>>     mount /mnt/btrtop/root
>>>     if [[ $? -ne 0 ]]; then \
>>>       echo "ERROR: failed to mount [/mnt/btrtop/root] (1 of 3)"
>>>     else
>>>       echo "OK: mounted [/mnt/btrtop/root] (1 of 3)"
>>>     fi
>>> else
>>>     echo "INFO: [/mnt/btrtop/root] (1 of 3) was already mounted. Nothing to do."
>>> fi
>>> findmnt /mnt/btrtop/home
>>> if [[ $? -ne 0 ]]; then \
>>>     mount /mnt/btrtop/home
>>>     if [[ $? -ne 0 ]]; then \
>>>       echo "ERROR: failed to mount [/mnt/btrtop/home] (2 of 3)"
>>>     else
>>>       echo "OK: mounted [/mnt/btrtop/home] (2 of 3)"
>>>     fi
>>> else
>>>     echo "INFO: [/mnt/btrtop/home] (2 of 3) was already mounted. Nothing to do."
>>> fi
>>> findmnt /mnt/btrtop/user
>>> if [[ $? -ne 0 ]]; then \
>>>     mount /mnt/btrtop/user
>>>     if [[ $? -ne 0 ]]; then \
>>>       echo "ERROR: failed to mount [/mnt/btrtop/user] (3 of 3)"
>>>     else
>>>       echo "OK: mounted mount [/mnt/btrtop/user] (3 of 3)"
>>>     fi
>>> else
>>>     echo "INFO: [/mnt/btrtop/user] (3 of 3) was already mounted. Nothing to do."
>>> fi
>>>
>>> echo "Finished mounting btrbk btrtop volumes."
>>>
>>> }
>>>
>>> btrbk_mount
>>>
>>> # end of file /usr/local/bin/btrbk_mount
>>>
>>>
>>>>
>>>>> Aug 06 03:00:14 btrbk_run.sh[3022708]: Finished mounting btrbk btrtop volumes.
>>>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: mounting btrbk btrtop volumes...
>>>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/root] (1 of
>>>>> 3) was already mounted. Nothing to do.
>>>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/home] (2 of
>>>>> 3) was already mounted. Nothing to do.
>>>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: INFO: [/mnt/btrtop/user] (3 of
>>>>> 3) was already mounted. Nothing to do.
>>>>> Aug 06 04:00:14 btrbk_run.sh[3033520]: Finished mounting btrbk btrtop volumes.
>>>>>
>>>>>>
>>>>>>>
>>>>>>> The path /var/cache/pacman is not a subvolume, but it resides on btrfs
>>>>>>> subvolume @btrtop/snapshot. @btrtop/snapshot is normally mounted at
>>>>>>> "/" but for btrfs tasks, it is also mounted at /mnt/btrtop/root. This
>>>>>>> additional mount operation seems to be causing these nested mounts of
>>>>>>> my bind mount for  /srv/nfs/var/cache/pacman .
>>>>>>>
>>>>>>> P.S. I cannot test without using systemd. (I'm not even sure I
>>>>>>> remember how to use a non-systemd distro anymore!)
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
>>>>>> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5
>>>>
>>>>
>>>> --
>>>> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
>>>> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5
>>
>>
>> --
>> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
>> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5


-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5