[RFC] online scrub for ext4

[RFC] online scrub for ext4

[Darrick J. Wong]

Hi all,

On this week's conference call, we discussed polishing e2croncheck into
something resembling a workable package, and I mentioned that some time
ago I'd adapted the script into a more general utility.  I've cleaned up
the scripts somewhat, so we can use them as a basis for discussion.

The first mail contains the base script to create the lvm snapshot, fsck
the snapshot, remove the snapshot, and fstrim the original filesystem.
It also contains a udev script to discourage it from making symlinks in
/dev/disk.

The second mail contains a script to identify eligible lvs containing
ext4 filesystms and call the actual invoker script.

The third mail contains systemd units to manage invocation and
containment of the scripts.

Note that despite my earlier outline, I don't actually have a boot time
script that cleans out any leftover fsck snapshots, though it wouldn't
be difficult to write.

Note also the lack of distro installer/partitioner support -- your ext4
filesystems /must/ be managed by lvm, and you /must/ have at least 256M
free in each volume group.  There's no daemon to watch for snapshots
filling up in order to kill the fsck and snapshot.

--D

[RFC 1/3] scripts to create and fsck a snapshot

[Darrick J. Wong]

/lib/udev/rules.d/96-online-fsck.rules:

# Try to hide our fsck snapshots from udev's /dev/disk linking...
ACTION=="add|change", ENV{DM_LV_NAME}=="*.fsck", OPTIONS="link_priority=-100"

/sbin/online-fsck:

#!/bin/bash

# Automatically check a LVM-managed filesystem online.
# We use lvm snapshots to do this, which means that we can only
# check filesystems in VGs that have at least 256mb (or so) of
# free space.
snap_size_mb=256
fstrim=0

print_help() {
	echo "Usage: $0 [-t] device"
	echo "-t: Run fstrim if successful."
}

exitcode() {
	ret="$1"
	if [ -n "${SERVICE_MODE}" ] && [ "${ret}" -ne 0 ]; then
		ret="$((ret + 150))"
	fi
	exit "${ret}"
}

while getopts "t" opt; do
	case "${opt}" in
	"t") fstrim=1;;
	*) print_help; exitcode 2;;
	esac
done
shift "$((OPTIND - 1))"

dev="$1"
if [ -z "${dev}" ]; then
	print_help
	exitcode 2
elif [ ! -b "${dev}" ]; then
	echo "${dev}: Not a block device?"
	print_help
	exitcode 2
fi

# Make sure this is an LVM device we can snapshot
lvs="$(lvs --noheadings -o vg_name,lv_name "${dev}" 2> /dev/null)"
if [ -z "${lvs}" ]; then
	echo "${dev}: Not a LVM device."
	exitcode 1
fi
vg="$(echo "${lvs}" | awk '{print $1}')"
lv="$(echo "${lvs}" | awk '{print $2}')"
start_time="$(date +'%Y%m%d%H%M%S')"
snap="${lv}.fsck"
snap_dev="/dev/${vg}/${snap}"
fstype="$(blkid -p -s TYPE "${dev}" | sed -e 's/^.*TYPE="\(.*\)".*$/\1/g')"

teardown() {
	lvremove -f "${vg}/${snap}" 3>&-
	while [ -b "${snap_dev}" ] && [ "$?" -eq "5" ]; do
		/bin/sleep 0.5
		lvremove -f "${vg}/${snap}" 3>&-
	done
}

check() {
	case "${fstype}" in
	"ext2"|"ext3"|"ext4")
		E2FSCK_FIXES_ONLY=1
		export E2FSCK_FIXES_ONLY
		opts="-vtt"
		${DBG} e2fsck -p ${opts} "${snap_dev}" || return 1
		${DBG} e2fsck -fy ${opts} "${snap_dev}" || return 1
		;;
	*)
		${DBG} fsck -n "${snap_dev}" || return 1
		;;
	esac
	return 0
}

mark_clean() {
	case "${fstype}" in
	"ext2"|"ext3"|"ext4")
		${DBG} tune2fs -C 0 -T "${start_time}" "${dev}"
		;;
	esac
}

mark_corrupt() {
	case "${fstype}" in
	"ext2"|"ext3"|"ext4")
		${DBG} tune2fs -C 16000 -T "19000101" "${dev}"
		;;
	esac
}

# Create the snapshot
echo "Scrubbing ${dev}."
teardown > /dev/null 2> /dev/null
trap "teardown" EXIT INT QUIT TERM
lvcreate -s -L "${snap_size_mb}m" -n "${snap}" "${vg}/${lv}" 3>&-
if [ $? -ne 0 ]; then
	echo "Snapshot of ${dev} FAILED, will not check!"
	exitcode 1
fi
udevadm settle

# Check and react
if check; then
	echo "Scrub of ${dev} succeeded."
	mark_clean

	if [ "${fstrim}" -eq 1 ]; then
		dir="$(lsblk -o MOUNTPOINT -n "${dev}")"
		if [ -d "${dir}" ]; then
			# NB: fstrim fails with snapshot present
			trap '' EXIT
			teardown
			fstrim -v "${dir}"
		fi
	fi

	ret=0
else
	echo "Scrub of ${dev} FAILED!  Reboot soon to fsck."
	mark_corrupt
	ret=2
fi

# Stupid journald bug where the process still has to exist for
# the last few messages to get tagged to the service...
if [ -n "${SERVICE_MODE}" ]; then
	sleep 2
fi

exitcode "${ret}"

[RFC 2/3] scripts to find eligible lv ext4 filesystems

[Darrick J. Wong]

/sbin/online-fsck-all:

#!/bin/bash

types="ext2,ext3,ext4"

# Scrub any fs on lvm by creating a snapshot and fscking that.
lvs --noheadings 2> /dev/null | while read lv vg junk; do
	dev="/dev/${vg}/${lv}"
	blkid -p -n "${types}" "${dev}" > /dev/null 2>&1 || continue

	${DBG} systemctl start "online-fsck@${dev}" 2> /dev/null
	res=$?
	if [ "${res}" -eq 0 ] || [ "${res}" -eq 1 ]; then
		if [ "${res}" -gt 150 ]; then
			res="$((res - 150))"
		fi
		echo "Scrubbing ${dev} done, (err=${res})"
	else
		${DBG} /sbin/online-fsck "${dev}"
	fi
done

# Stupid journald bug where the process still has to exist for
# the last few messages to get tagged to the service...
test -n "${SERVICE_MODE}" && sleep 2

exit 0

[RFC 3/3] systemd integration scripts

[Darrick J. Wong]

Here's the systemd stuff.  I haven't bothered to supply a cronjob,
since it's trivial to create a cron.weekly entry that runs
/sbin/online-fsck-all.

--D

/lib/systemd/system/online-fsck-all.service:

[Unit]
Description=Scrub Filesystem Metadata
ConditionACPower=true
 
[Service]
Type=oneshot
Environment=SERVICE_MODE=1
ExecStart=/sbin/online-fsck-all

/lib/systemd/system/online-fsck-all.timer:

[Unit]
Description=Scrub Filesystem Metadata Periodically

[Timer]
# Run on Sunday at 3:30am, to avoid running afoul of DST changes
OnCalendar=Sun *-*-* 03:30:00
RandomizedDelaySec=60
Persistent=true

[Install]
WantedBy=timers.target

/lib/systemd/system/online-fsck-fail@.service:

[Unit]
Description=Scrub Filesystem Metadata Failure Reporting for %I

[Service]
Type=oneshot
Environment=EMAIL_ADDR=root@localhost
ExecStart=/usr/lib/online-fsck/online-scrub-fail "${EMAIL_ADDR}" %I
User=mail
Group=mail
SupplementaryGroups=systemd-journal

/lib/systemd/system/online-fsck@.service:

[Unit]
Description=Scrub Filesystem Metadata for %I
OnFailure=online-fsck-fail@%i.service

[Service]
Type=oneshot
WorkingDirectory=/
PrivateNetwork=true
ProtectSystem=true
ProtectHome=read-only
PrivateTmp=yes
AmbientCapabilities=CAP_SYS_ADMIN CAP_SYS_RAWIO
NoNewPrivileges=yes
User=root
IOSchedulingClass=idle
CPUSchedulingPolicy=idle
Environment=SERVICE_MODE=1
ExecStart=/sbin/online-fsck -t %I

/usr/lib/online-fsck/online-scrub-fail:

#!/bin/bash

# Email logs of failed online-fsck unit runs

mailer=/usr/sbin/sendmail
recipient="$1"
test -z "${recipient}" && exit 0
dev="$2"
test -z "${dev}" && exit 0
hostname="$(hostname -f 2>/dev/null)"
test -z "${hostname}" && hostname="${HOSTNAME}"
if [ ! -x "${mailer}" ]; then
	echo "${mailer}: Mailer program not found."
	exit 1
fi

(cat << ENDL
To: $1
From: <online-fsck@${hostname}>
Subject: online-fsck failure on ${dev}

So sorry, the automatic online-fsck of ${dev} on ${hostname} failed.

A log of what happened follows:
ENDL
systemctl status --full --lines 4294967295 "online-fsck@${dev}") | "${mailer}" -t -i