[Buildroot] Obscure bug in Util-Linux umount?

[Buildroot] Obscure bug in Util-Linux umount?

[Steve Kenton]

I *think* this is an obscure bug in the util-linux version of umount,
but I wonder if anyone else has run into it. It drove me nuts for a while.

Here's a short description.

I am developing an x86 imbedded system using buildroot-2015.02 running
on an Ubuntu 14.10 host. I have two ssd drives formatted as ext4 and
mounted on mount point directories /ssd (/dev/sda1) and /data (/dev/sdb1)
on the Ubuntu system whose root filesystem / (/dev/sdc1) is selected through
the bios at power up as the boot device.

This allows me to boot and run the embedded system in it's 'natural' place
using /dev/sda and /dev/sdb during development and testing in an effort to
keep life simple. According to "dumpe2fs -h" the superblocks of /dev/sda1
and /dev/sdb1 contain the last mounted locations of /ssd and /data as expected
when booting the embedded system.

Here is the odd thing that occurs when running the embedded system.
If the embedded root filesystem / has a mount point directory
*with the same name* as appears in it's superblock then anything mounted
on that mount point will always be "busy" and cannot be unmounted using
the util-linux versions of umount, but unmounts just fine with the busybox
version of umount. Mounting and unmounting on any other mount point directory
causes no problems.

In my case where '/ssd' (/dev/sda1) is the root filesystem and '/data' (/dev/sdb1)
is for data, mounting /dev/sdb1 on a /data directory located on the emedded root /dev/sda1
acts as expected but mounting /dev/sdb1 on a /ssd directory located on the embedded
root /dev/sda1 results in a "busy" filesystem that cannot be unmounted using util-linux
umount but can be unmounted using busybox umount. Switching the root and data disk functions
causes the problem to move to the other disk so it's not tied to a specific mount point
directory name.

It's easy enough to work around if you know about it but I would not wish the
debugging on anyone else.

So, is this a feature, a bug, or am I just too dumb for words and missing something obvious?

Steve

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20150310/d5391e39/attachment.html>

[Buildroot] Obscure bug in Util-Linux umount?

[Steve Kenton]

On 03/10/2015 12:18 AM, Steve Kenton wrote:
> I *think* this is an obscure bug in the util-linux version of umount, but I wonder if anyone else has run into it. It drove me nuts for a while. Here's a short description.

Further testing revealed that it only occurs if the embedded root filesystem is read-only
which I forgot to mention (blush) so it's probably an interaction with a stale /etc/mtab
or something, but only for the root filesystem and only if the mount point directory names
are the same, yada etc. Much less likely to bite anyone else so I guess a heads up is not needed.

Any way, not really a bug but rather a feature with  undefined behaviour and most importantly
a case of 'it hurts if I do this' soooo  'don't do that'!

Sorry for the noise.

Steve

[Buildroot] Obscure bug in Util-Linux umount?

[Thomas Petazzoni]

Dear Steve Kenton,

On Tue, 10 Mar 2015 11:16:20 -0500, Steve Kenton wrote:

> Further testing revealed that it only occurs if the embedded root filesystem is read-only
> which I forgot to mention (blush) so it's probably an interaction with a stale /etc/mtab
> or something, but only for the root filesystem and only if the mount point directory names
> are the same, yada etc. Much less likely to bite anyone else so I guess a heads up is not needed.
> 
> Any way, not really a bug but rather a feature with  undefined behaviour and most importantly
> a case of 'it hurts if I do this' soooo  'don't do that'!

Interesting. After reading your e-mail, I had a very quick look at the
kernel side and util-linux side, looking for code using the
s_last_mounted field of the superblock which indeed contains the last
location where it was mounted. But I couldn't spot any place where it
is specifically used to prevent something from happening. But I didn't
spend a lot of time, so I may very well have missed something.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Obscure bug in Util-Linux umount?

[Steve Kenton]

On 3/10/2015 12:59 PM, Thomas Petazzoni wrote:
> Dear Steve Kenton,
>
> On Tue, 10 Mar 2015 11:16:20 -0500, Steve Kenton wrote:
>
>> Further testing revealed that it only occurs if the embedded root filesystem is read-only
>> which I forgot to mention (blush) so it's probably an interaction with a stale /etc/mtab
>> or something, but only for the root filesystem and only if the mount point directory names
>> are the same, yada etc. Much less likely to bite anyone else so I guess a heads up is not needed.
>>
>> Any way, not really a bug but rather a feature with  undefined behaviour and most importantly
>> a case of 'it hurts if I do this' soooo  'don't do that'!
> Interesting. After reading your e-mail, I had a very quick look at the
> kernel side and util-linux side, looking for code using the
> s_last_mounted field of the superblock which indeed contains the last
> location where it was mounted. But I couldn't spot any place where it
> is specifically used to prevent something from happening. But I didn't
> spend a lot of time, so I may very well have missed something.
>
> Thomas
If anyone else has ever hit this then it might be worth pursuing and 
I'll take another look,
but if nobody else cares I'll just chock it up to dumb-(bad)-luck and 
move along.

Any one out there care about this? It's easy for me to reproduce right 
now while (painfully) fresh in my memory.

Steve

[Buildroot] Obscure bug in Util-Linux umount? - Solved

[Steve Kenton]

On 03/10/2015 01:05 PM, Steve Kenton wrote:
> On 3/10/2015 12:59 PM, Thomas Petazzoni wrote:
>> Dear Steve Kenton,
>>
>> On Tue, 10 Mar 2015 11:16:20 -0500, Steve Kenton wrote:
>>
>>> Further testing revealed that it only occurs if the embedded root filesystem is read-only
>>> which I forgot to mention (blush) so it's probably an interaction with a stale /etc/mtab
>>> or something, but only for the root filesystem and only if the mount point directory names
>>> are the same, yada etc. Much less likely to bite anyone else so I guess a heads up is not needed.

I couldn't leave it alone :-) And, it turns out I was pretty far off in some of my original ideas.
The link to "last mounted on" turned out to be an artifact of how I was testing and does not in fact matter.
The link to "read only" was because I could initially only reproduce using a bootable DVD and does not in fact matter.

The link to "mount point directory" having the same name on both host and target systems however was spot on.

The script below can reproduce the problem but it take 3 separate disks and pretty much the exact oddball
file layout that I was using to create the bootable 'recovery' DVD before it can trigger. I would not be
surprised if nobody else has ever run across this particular problem combination before.

That said, the fact that glibc binaries contain an embedded reference to the host build location no matter
where they are ultimately installed could possibly be a problem for others at some point. I for sure was not
aware of it. Read below for more details. It's definitely not specific to just umount though.

Steve


#!/bin/bash
# Reproduce the filesystem "busy" umount problem with util-linux umount but not busybox umount
# Glibc uses shared libraries and building glibc util-linux causes the umount binary to contain a reference to the host build location
# Which in my case was a disk mounted at /data on an Ubuntu 14.04 system where I was building buildroot
# When booting the resulting buildroot system which I installed onto a disk mounted on /ssd the following output was obtained

build=/data
target=/ssd

# which umount
#   /bin/umount
# strings /bin/umount | grep /data
#   /data/buildroot/output/build/util-linux-2.26/.libs

# Apparently this embedded reference is the preferred location on-disk and is checked before anything else
# So, when we mount the build filesystem on a same-named mount point directory on the running target system
# attempting to then umount it fails because the umount command itself is using those libs on the build filesystem
# but as soon as mount exits that use ends and lsof and fuser will not see anything in-use on the build filesystem

# This script reproduces the problem by building a similar system using the i386 minimal configuration embedded in a here-document below
#
# Notice that the default buildroot grub2 assumption is currently MBR (hd0,msdos1) while I'm using GPT (hd0,gpt1) - adjust accordingly for your setup
#
# The build and target mount point directories must be a immediately off the root and must have filesystems (in partition #1) mounted on them
# On the build filesystem we create 'buildroot' which is the buildroot snapshot directory name
# The target filesystem is made bootable and any previously existing system will be over written
if [ $# -lt 2 ]; then
	echo "Usage: $0 <build_mount_point> <target_mount_point>"
	echo "You must be able to sudo and understand the risks of doing so if you run this"
	echo "Read the comments in the script for more details"
	exit 0
fi

# Get the last line of POSIX df output using tail, keep the first space delimited field using cut and delete the last character using sed
rootdev=$(df -P / | tail -n 1 | cut -d' ' -f 1 | sed 's/.$//') # the base (unpartitioned) host root file system device
buildon=$(df -P $build | tail -n 1 | cut -d' ' -f 1 | sed 's/.$//') # the base (unpartitioned) build file system device
installto=$(df -P $target | tail -n 1 | cut -d' ' -f 1 | sed 's/.$//') # the base (unpartitioned) target file system device where GRUB2 will be installed
# A bit of sanity checking
if [ "$installto" == "$rootdev" ]; then
	echo "WRITING OVER YOUR RUNNING ROOT FILE SYSTEM IS A BAD IDEA!"
	exit 1
fi
# Make sure we have buildroot
if [ ! -d $build/buildroot ]; then
	sudo chown $user:$user $build
	mkdir $build/buildroot
	cd $build
	if [ ! -f buildroot-snapshot.tar.bz2 ]; then
		wget http://buildroot.org/downloads/buildroot-snapshot.tar.bz2
	fi
	tar --extract --file buildroot-snapshot.tar.bz2 # tar will automatically figure out the compression
fi
cd $build/buildroot
if [ "$(pwd)" != "$build/buildroot" ]; then
	echo "Can not cd to $build/buildroot"
	exit 1
fi
# The configuration to use
cat > .config <<'EOF'
BR2_TOOLCHAIN_BUILDROOT_GLIBC=y
BR2_LINUX_KERNEL=y
BR2_LINUX_KERNEL_DEFCONFIG="i386"
BR2_LINUX_KERNEL_INSTALL_TARGET=y
BR2_PACKAGE_UTIL_LINUX=y
BR2_PACKAGE_UTIL_LINUX_BINARIES=y
BR2_PACKAGE_UTIL_LINUX_MOUNT=y
BR2_TARGET_GRUB2=y
BR2_TARGET_GRUB2_BOOT_PARTITION="hd0,gpt1"
EOF
make olddefconfig # fill out the minimal config with default answers
make # and get to work

# A quick summary
ls -ld $build $target
mount | grep /dev/sd | sort
echo "Grub2 and buildroot need to be installed on the target system boot/root filesystem disk to create a bootable system"
read -p "Do you want delete and repopulate buildroot on the target system root filesystem $target (${installto}1) and install Grub2 to make $installto bootable? (yes or no) "
if [ "$REPLY" == "yes" ]; then
	(cd $target; sudo rm -rf bin boot dev etc home init lib lib64 libexec linuxrc media mnt opt proc root run sbin sys tmp usr var) # subshell
	sudo tar --extract --file output/images/rootfs.tar --directory $target # the buildroot root filesystem
	sudo output/host/usr/sbin/grub-bios-setup --boot-image=output/host/usr/lib/grub/i386-pc/boot.img --core-image=output/images/grub.img --directory=. $installto # paths relative to -d
fi
sudo mkdir -p $target$build # this is the poison pill - a mount point directory on the target with the same name as on the build mount point directory on the build system
sudo ln -s /bin/busybox $target/root/umount # root user's home directory on buildroot
echo "Boot the system now installed on $target (${installto}1) and enter the following commands"
echo "mount ${buildon}1 $build"
echo "umount $build" # util-linux umount
echo "./umount $build" # busybox umount
echo "The newly mounted $build filesystem should be 'busy' and not umountable using util-linux umount but should unmount ok with busybox umount"
# I'm guessing that the reason busybox umount works is either that it's statically linked or it's
# dynamically linked but the libraries are already in memory because of other busybox applets running