[LTP] lib: tst_device.c - /dev/loop0 in use

[LTP] lib: tst_device.c - /dev/loop0 in use

[Rafael David Tinoco]

Hello Cyril,

Apparently, when having "all_filesystems" flag set for one test, there
is a room of opportunity for the subsequent filesystems tests to find
the device, acquired by "tst_acquire_device()", busy.

I suppose, in the case bellow, fsync() caused async work in some kernel
thread to cause the "/dev/loop0 is apparently in use by the system"
error when trying a new mkfs.XXXX call - for the subsequent filesystem -
using the same acquired device.

With the "all_filesystems" logic, the acquired device is not detached,
nor find_free_loopdev() is called again, which *could* mitigate this.

The error:

tst_test.c:1015: INFO: Timeout per run is 0h 15m 00s
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
tst_test.c:1076: INFO: Testing on ext3
tst_mkfs.c:83: INFO: Formatting /dev/loop0 with ext3 opts='' extra opts=''
mke2fs 1.43.5 (04-Aug-2017)
[   48.457646] EXT4-fs (loop0): mounting ext3 file system using the ext4
subsystem
[   48.502872] EXT4-fs (loop0): mounted filesystem with ordered data
mode. Opts: (null)
tst_test.c:1015: INFO: Timeout per run is 0h 15m 00s
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
fsync01.c:58: PASS: fsync() returned 0
tst_test.c:1076: INFO: Testing on ext4
tst_mkfs.c:83: INFO: Formatting /dev/loop0 with ext4 opts='' extra opts=''
mke2fs 1.43.5 (04-Aug-2017)
/dev/loop0 is apparently in use by the system; will not make a
filesystem here!
tst_mkfs.c:94: BROK: mkfs.ext4:1: tst_test.c failed with 729

Thoughts ?

Cheers o/
-- 
Rafael D. Tinoco
Linaro - Kernel Validation

[LTP] lib: tst_device.c - /dev/loop0 in use

[Cyril Hrubis]

Hi!
> Apparently, when having "all_filesystems" flag set for one test, there
> is a room of opportunity for the subsequent filesystems tests to find
> the device, acquired by "tst_acquire_device()", busy.
> 
> I suppose, in the case bellow, fsync() caused async work in some kernel
> thread to cause the "/dev/loop0 is apparently in use by the system"
> error when trying a new mkfs.XXXX call - for the subsequent filesystem -
> using the same acquired device.
> 
> With the "all_filesystems" logic, the acquired device is not detached,
> nor find_free_loopdev() is called again, which *could* mitigate this.

Right, we expect that the device is free to be reused once the umount()
returns to the userspace, which apparently is not true in your case.

Looking into the mkfs code it tries to open() the device RDONLY and we
get this error if the attempt has failed with EBUSY, which likely means
that the umount() is still running in background in kernel.

I guess that we may as well patch the tst_umount() function in
lib/tst_device.c to loop until open() fails with EBUSY in the similar
manner we attempt to retry umount() there.

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] [PATCH] tst_umount: Retry open() after umount to handle delayed device release

[Li Wang via ltp]

Currently, tst_umount() in lib/tst_device.c tries to repeatedly umount()
a mount point, with a retry loop if it gets EBUSY. However, after umount()
reports success, devices (especially loop devices) can still be held open
by another process or kernel, delaying their actual release. This can lead
to race conditions when the next operation tries to reuse the device like
mkfs.ext3 error:

  ==== setxattr01 ====
  command: setxattr01
  tst_test.c:1953: TINFO: LTP version: 20250530
  ...
  tst_supported_fs_types.c:48: TINFO: mkfs is not needed for tmpfs
  tst_test.c:1888: TINFO: === Testing on ext2 ===
  tst_test.c:1217: TINFO: Formatting /dev/loop0 with ext2 opts='' extra opts=''
  mke2fs 1.47.1 (20-May-2024)
  tst_test.c:1229: TINFO: Mounting /dev/loop0 to /tmp/LTP_setd24dAf/mntpoint fstyp=ext2 flags=0
  ...
  setxattr01.c:174: TPASS: setxattr(2) failed: ERANGE (34)
  setxattr01.c:174: TPASS: setxattr(2) failed: EFAULT (14)
  tst_test.c:1888: TINFO: === Testing on ext3 ===
  tst_test.c:1217: TINFO: Formatting /dev/loop0 with ext3 opts='' extra opts='' mke2fs 1.47.1 (20-May-2024)

  /dev/loop0 is apparently in use by the system; will not make a filesystem here!
  tst_test.c:1217: TBROK: mkfs.ext3 failed with exit code 1

  ==== close_range01 ====
  command: close_range01
  tst_test.c:1953: TINFO: LTP version: 20250530
  ...
  tst_test.c:1888: TINFO: === Testing on ext2 ===
  tst_test.c:1217: TINFO: Formatting /dev/loop0 with ext2 opts='' extra opts=''
  mke2fs 1.47.1 (20-May-2024)
  tst_test.c:1229: TINFO: Mounting /dev/loop0 to /tmp/LTP_cloXeXI39/mnt fstyp=ext2 flags=0
  ...
  close_range01.c:188: TPASS: No kernel taints
  tst_test.c:1888: TINFO: === Testing on ext3 ===
  tst_test.c:1217: TINFO: Formatting /dev/loop0 with ext3 opts='' extra opts='' mke2fs 1.47.1 (20-May-2024)

  /dev/loop0 is apparently in use by the system; will not make a filesystem here!
  tst_test.c:1217: TBROK: mkfs.ext3 failed with exit code 1

This patch improves robustness by, after a successful umount(), attempting to
open() the device in O_RDWR mode and retrying if open() fails with EBUSY,
similar to how umount() is retried. This ensures that tst_umount() only returns
success after the device is truly available for reuse, reducing race conditions
in rapid test cycles.

Signed-off-by: Li Wang <liwang@redhat.com>
---
 lib/tst_device.c | 27 +++++++++++++++++++++++++--
 1 file changed, 25 insertions(+), 2 deletions(-)

diff --git a/lib/tst_device.c b/lib/tst_device.c
index 6d1abf065..7f0a4f8a4 100644
--- a/lib/tst_device.c
+++ b/lib/tst_device.c
@@ -397,14 +397,37 @@ int tst_clear_device(const char *dev)
 
 int tst_umount(const char *path)
 {
-	int err, ret, i;
+	int err, ret, i, j;
 
 	for (i = 0; i < 50; i++) {
 		ret = umount(path);
 		err = errno;
 
-		if (!ret)
+		if (!ret) {
+			/* After successful umount, check device open busy */
+			for (j = 0; j < 50; j++) {
+				int fd = open(path, O_RDWR);
+				if (fd >= 0) {
+					close(fd);
+					break;
+				}
+				if (errno == EBUSY) {
+					tst_resm(TINFO, "open('%s') got EBUSY after umount, retry %d...",
+						path, j+1);
+					usleep(100000);
+					continue;
+				}
+				break;
+			}
+			if (j == 50) {
+				tst_resm(TWARN, "Device '%s' is still busy after %d open retries",
+					path, 10);
+				errno = EBUSY;
+				return -1;
+			}
+
 			return 0;
+		}
 
 		if (err != EBUSY) {
 			tst_resm(TWARN, "umount('%s') failed with %s",
-- 
2.50.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH] tst_umount: Retry open() after umount to handle delayed device release

[Cyril Hrubis]

Hi!
> Currently, tst_umount() in lib/tst_device.c tries to repeatedly umount()
> a mount point, with a retry loop if it gets EBUSY. However, after umount()
> reports success, devices (especially loop devices) can still be held open
> by another process or kernel, delaying their actual release. This can lead
> to race conditions when the next operation tries to reuse the device like
> mkfs.ext3 error:

This looks like working around a kernel bug to me. The umount() without
MNT_DETACH should make the device reusable once the syscall returns back
to the userspace. At least I wouldn't add this code to LTP unless kernel
devs tells us that this broken behavior is to be expected.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH] tst_umount: Retry open() after umount to handle delayed device release

[Li Wang via ltp]

Hi Cyril,

On Wed, Aug 6, 2025 at 5:26 PM Cyril Hrubis <chrubis@suse.cz> wrote:

> Hi!
> > Currently, tst_umount() in lib/tst_device.c tries to repeatedly umount()
> > a mount point, with a retry loop if it gets EBUSY. However, after
> umount()
> > reports success, devices (especially loop devices) can still be held open
> > by another process or kernel, delaying their actual release. This can
> lead
> > to race conditions when the next operation tries to reuse the device like
> > mkfs.ext3 error:
>
> This looks like working around a kernel bug to me. The umount() without
> MNT_DETACH should make the device reusable once the syscall returns back
> to the userspace. At least I wouldn't add this code to LTP unless kernel
> devs tells us that this broken behavior is to be expected.


Perhaps not yet fall into the kernel issue.

Looking into the details for many days, there seems to be something in
userspace briefly reopens the block device right after unmounting it,
I suspect that “something” is likely systemd-udevd running its blkid
builtin.

To prove this, I did some debugging work:

1. Confirm the tested system enables rule:
# cat /usr/lib/udev/rules.d/60-persistent-storage.rules | grep blkid
  KERNEL!="sr*|mmcblk[0-9]boot[0-9]", IMPORT{builtin}="blkid"

2. Check the core functions of udev-builtin-blkid:

https://github.com/systemd/systemd/blob/main/src/udev/udev-builtin-blkid.c#L353

3. Run udev monitor in a separate terminal, and get output with running
./close_range01 test simultaneously:

# udevadm monitor --kernel --udev
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent

KERNEL[283886.216245] change   /devices/virtual/block/loop0 (block)
KERNEL[283886.271087] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.277479] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.280656] change   /devices/virtual/block/loop0 (block)
KERNEL[283886.299308] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.303885] change   /devices/virtual/block/loop0 (block)
KERNEL[283886.307296] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.313293] change   /devices/virtual/block/loop0 (block)
KERNEL[283886.317716] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.321284] change   /devices/virtual/block/loop0 (block)
KERNEL[283886.358952] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.364841] change   /devices/virtual/block/loop0 (block)
KERNEL[283886.369347] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.376379] change   /devices/virtual/block/loop0 (block)
KERNEL[283886.378823] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.382704] change   /devices/virtual/block/loop0 (block)
KERNEL[283886.463900] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.467966] change   /devices/virtual/block/loop0 (block)
KERNEL[283886.471508] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.475901] change   /devices/virtual/block/loop0 (block)
KERNEL[283886.491129] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.495081] change   /devices/virtual/block/loop0 (block)
KERNEL[283886.498525] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.502380] change   /devices/virtual/block/loop0 (block)
KERNEL[283886.504077] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.507968] change   /devices/virtual/block/loop0 (block)
KERNEL[283886.558605] change   /devices/virtual/block/loop0 (block)
KERNEL[283886.559145] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.560932] change   /devices/virtual/block/loop0 (block)
UDEV  [283886.562050] change   /devices/virtual/block/loop0 (block)

Looks like udev-builtin-blkid indeed probes via libblkid to populate
ID_FS_* properties.

Then, I use strace tool to trace the openat on systemd-udevd, but weirdly
I got all opens with 'O_PATH',  which opens do not conflict with mkfs.ext*’s
O_EXCL and should not cause “apparently in use by the system”.

4. #sudo strace -f -e trace=openat,close -p $(pidof systemd-udevd) -o log
# cat log | grep '/dev/loop0'
275485 openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH)
= 17
702   openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH
<unfinished ...>
275485 openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH)
= 17
275485 openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_CLOEXEC|O_PATH) = 17
275485 openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH)
= 17
275485 openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH)
= 17
275485 openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_CLOEXEC|O_PATH) = 17
702   openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH)
= 13
275485 openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH)
= 17
...

However, switch to another bpftrace tool gives more suggestive clues,
it catches not only the udev-worker but also udeskd operations on the
/dev/loop0, after checking the flags I doubt that race probably comes
from udiskd operation.

5. Run bpftrace in a separate terminal and get output with running
./close_range01 test simultaneously:

# sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat
/str(args->filename)=="/dev/loop0"/ { printf("%s pid=%d flags=0x%x %s\n",
comm, pid, args->flags, str(args->filename)); }'
Attaching 1 probe...
close_range01 pid=298948 flags=0x2 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
close_range01 pid=298948 flags=0x0 /dev/loop0
systemd-udevd pid=702 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x280000 /dev/loop0
udisksd pid=87323 flags=0x0 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x280000 /dev/loop0
udisksd pid=87323 flags=0x0 /dev/loop0
close_range01 pid=298948 flags=0x241 /dev/loop0
systemd-udevd pid=702 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
mkfs.ext2 pid=298969 flags=0x80 /dev/loop0
mkfs.ext2 pid=298969 flags=0x0 /dev/loop0
mkfs.ext2 pid=298969 flags=0x0 /dev/loop0
mkfs.ext2 pid=298969 flags=0x0 /dev/loop0
mkfs.ext2 pid=298969 flags=0x80800 /dev/loop0
mkfs.ext2 pid=298969 flags=0x0 /dev/loop0
mkfs.ext2 pid=298969 flags=0x82 /dev/loop0
(udev-worker) pid=298949 flags=0x280000 /dev/loop0
udisksd pid=87323 flags=0x0 /dev/loop0
systemd-udevd pid=702 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
udisksd pid=87323 flags=0x0 /dev/loop0
(udev-worker) pid=298949 flags=0x280000 /dev/loop0
udisksd pid=87323 flags=0x0 /dev/loop0
udisksd pid=87323 flags=0x0 /dev/loop0        <--------------
close_range01 pid=298948 flags=0x241 /dev/loop0
systemd-udevd pid=702 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
(udev-worker) pid=298949 flags=0x288000 /dev/loop0
mkfs.ext3 pid=298977 flags=0x80 /dev/loop0
mkfs.ext3 pid=298977 flags=0x0 /dev/loop0
mkfs.ext3 pid=298977 flags=0x0 /dev/loop0
mkfs.ext3 pid=298977 flags=0x0 /dev/loop0
mkfs.ext3 pid=298977 flags=0x80800 /dev/loop0
mkfs.ext3 pid=298977 flags=0x0 /dev/loop0
mkfs.ext3 pid=298977 flags=0x82 /dev/loop0
...

From the bpfstrace logs, there seems udisksd listening to the same
udevevents and probes the device via libblockdev/blkid, performing
real O_RDONLY opens (flags=0x0). These O_RDONLY opens from
udisksd race with mkfs.ext3’s O_EXCL open, intermittently causing
“apparently in use by the system” (EBUSY).

Does this analysis sound reasonable to you?

-- 
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH] tst_umount: Retry open() after umount to handle delayed device release

[Cyril Hrubis]

Hi!
> From the bpfstrace logs, there seems udisksd listening to the same
> udevevents and probes the device via libblockdev/blkid, performing
> real O_RDONLY opens (flags=0x0). These O_RDONLY opens from
> udisksd race with mkfs.ext3’s O_EXCL open, intermittently causing
> “apparently in use by the system” (EBUSY).
> 
> Does this analysis sound reasonable to you?

Yes. So we have another daemon that probes block devices on each change.

I suppose that turning off udisksd fixes the problem right? If that is
so I would be inclined just to print a message to turn off udisksd if
mkfs on a loop device fails...

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH] tst_umount: Retry open() after umount to handle delayed device release

[Li Wang via ltp]

On Mon, Aug 11, 2025 at 10:19 PM Cyril Hrubis <chrubis@suse.cz> wrote:

> Hi!
> > From the bpfstrace logs, there seems udisksd listening to the same
> > udevevents and probes the device via libblockdev/blkid, performing
> > real O_RDONLY opens (flags=0x0). These O_RDONLY opens from
> > udisksd race with mkfs.ext3’s O_EXCL open, intermittently causing
> > “apparently in use by the system” (EBUSY).
> >
> > Does this analysis sound reasonable to you?
>
> Yes. So we have another daemon that probes block devices on each change.
>
> I suppose that turning off udisksd fixes the problem right? If that is
> so I would be inclined just to print a message to turn off udisksd if
> mkfs on a loop device fails...
>

That's right, this intermittent EBUSY is not related to LTP (and not
be a big deal), a hint print should be useful enough, we can leave
stopping udisks2 operation to the tester env setup.

What about this amendment:

--- a/lib/tst_mkfs.c
+++ b/lib/tst_mkfs.c
@@ -107,6 +107,13 @@ void tst_mkfs_(const char *file, const int lineno,
void (cleanup_fn)(void),
                        "%s not found in $PATH", mkfs);
        break;
        default:
+               tst_resm_(file, lineno, TINFO,
+                       "Check if mkfs failed with the (loop) device busy. "
+                       "Background probing (e.g., udisks2) can cause this.
\n"
+                       "Consider temporarily stopping udisks2 during the
test:\n"
+                       "  systemctl stop udisks2.service\n"
+                       "and restart afterward:\n"
+                       "  systemctl start udisks2.service");
                tst_brkm_(file, lineno, TBROK, cleanup_fn,
                        "%s failed with exit code %i", mkfs, ret);
        }


-- 
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH] tst_umount: Retry open() after umount to handle delayed device release

[Cyril Hrubis]

Hi!
> --- a/lib/tst_mkfs.c
> +++ b/lib/tst_mkfs.c
> @@ -107,6 +107,13 @@ void tst_mkfs_(const char *file, const int lineno,
> void (cleanup_fn)(void),
>                         "%s not found in $PATH", mkfs);
>         break;
>         default:
> +               tst_resm_(file, lineno, TINFO,
> +                       "Check if mkfs failed with the (loop) device busy. "
> +                       "Background probing (e.g., udisks2) can cause this.
> \n"
> +                       "Consider temporarily stopping udisks2 during the
> test:\n"
> +                       "  systemctl stop udisks2.service\n"
> +                       "and restart afterward:\n"
> +                       "  systemctl start udisks2.service");

I wouldn't include the systemctl commands in the message, anybody who is
using LTP to test the system should know how to start and stop system
daemons. Let's keep the message short and to the point.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp