Re: [PATCH v2 2/2] ext4/036: Add tests for filename casefolding feature

[Eryu Guan <guaneryu@gmail.com>]

On Thu, Jun 06, 2019 at 03:31:38PM -0400, Gabriel Krisman Bertazi wrote:
> From: "Lakshmipathi.G" <lakshmipathi.ganapathi@collabora.co.uk>
> 
> This new test implements verification for the per-directory
> case-insensitive feature, as supported in the reference implementation
> in Ext4.  Currently, this test only supports Ext4, but the plan is to
> run it in other filesystems, once they support the feature.
> 
> For now, let it live in ext4 and we move to shared/ or generic/ when
> other filesystems supporting this feature start to pop up.
> 
> Signed-off-by: Lakshmipathi.G <lakshmipathi.ganapathi@collabora.co.uk>
> Signed-off-by: Gabriel Krisman Bertazi <krisman@collabora.com>
>   [Rewrite to support feature design]
>   [Refactor to simplify implementation]
> ---
>  tests/ext4/036     | 469 +++++++++++++++++++++++++++++++++++++++++++++
>  tests/ext4/036.out |   2 +
>  tests/ext4/group   |   1 +
>  3 files changed, 472 insertions(+)
>  create mode 100755 tests/ext4/036
>  create mode 100644 tests/ext4/036.out
> 
> diff --git a/tests/ext4/036 b/tests/ext4/036
> new file mode 100755
> index 000000000000..3dae4212853b
> --- /dev/null
> +++ b/tests/ext4/036
> @@ -0,0 +1,469 @@
> +#! /bin/bash
> +# FSQA Test No. 036
> +#
> +# Test the creation, mounting and basic functionality of file name
> +# casefolding on Ext4
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2018 Collabora Ltd.  All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#
> +#-----------------------------------------------------------------------

Same here, use SPDX tags.

> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +status=1 # failure is thea default
> +
> +. ./common/rc
> +. ./common/casefold
> +
> +_supported_os Linux
> +_require_scratch
> +_require_casefold_support
> +_require_check_dmesg
> +
> +sdev=$(_short_dev ${SCRATCH_DEV})
> +test_casefold=${SCRATCH_MNT}

This doesn't look necessary, just use $SCRATCH_MNT in the test?

> +
> +filename1="file.txt"
> +filename2="FILE.TXT"
> +
> +pt_file1=$(echo -e "coração")
> +pt_file2=$(echo -e "corac\xcc\xa7\xc3\xa3o" | tr a-z A-Z)
> +
> +fr_file2=$(echo -e "french_caf\xc3\xa9.txt")
> +fr_file1=$(echo -e "french_cafe\xcc\x81.txt")
> +
> +ar_file1=$(echo -e "arabic_\xdb\x92\xd9\x94.txt")
> +ar_file2=$(echo -e "arabic_\xdb\x93.txt" | tr a-z A-Z)
> +
> +jp_file1=$(echo -e "japanese_\xe3\x82\xb2.txt")
> +jp_file2=$(echo -e "japanese_\xe3\x82\xb1\xe3\x82\x99.txt")
> +
> +# '\xc3\x00' is an invalid sequence. Despite that, the sequences below
> +# could match, if we ignored the error.  But we don't want to be greedy
> +# at normalization, so at the first error we treat the entire sequence
> +# as an opaque blob.  Therefore, these two must NOT match.
> +blob_file1=$(echo -e "corac\xcc\xa7\xc3")
> +blob_file2=$(echo -e "coraç\xc3")
> +
> +# Test helpers
> +basic_create_lookup() {
> +    touch "${basedir}/${1}"

Please assign one of the function params to basedir and declare it as a
local variable.

> +    stat  "${basedir}/${2}" &> /dev/null || \
> +	_fail "Casefolded lookup of ${1} ${2} failed"

In fstests, we usually don't do _fail in tests, we just let test prints
error messages and the test harness will capture the outputs & compare
them with the golden image (.out file), so we know which part of the
test fails and test could continue to run. e.g.

> +    _casefold_check_exact_name "${basedir}" "${1}" || \
> +	_fail "Created file with wrong name."

	stat "$basedir/$2" | _filter_scratch
	_casefold_check_exact_name "$basedir" "$1" || \
		echo "Created \"$basedir/$2\" with wrong name"

To use _filter_scratch we need to source common/filter after sourcing
common/rc.

> +
> +}
> +
> +# CI search should fail.
> +bad_basic_create_lookup() {
> +    touch "${basedir}/${1}"
> +    stat  "${basedir}/${2}" &> /dev/null && \
> +	_fail "Lookup of ${1} using ${2} should have failed"
> +}
> +
> +# Testcases
> +test_casefold_lookup () {
                      ^^^ no space here.
> +    basedir=${test_casefold}/casefold_lookup
> +    mkdir -p ${basedir}
> +
> +    _set_casefold_attr ${basedir}

Same here, if _set_casefold_attr fails with some error messages to
stderr, the test harness would capture that, just put the expected
output to .out file if any.

> +
> +    basic_create_lookup "${filename1}" "${filename2}"
> +    basic_create_lookup "${pt_file1}" "${pt_file2}"
> +    basic_create_lookup "${fr_file1}" "${fr_file2}"
> +    basic_create_lookup "${ar_file1}" "${ar_file2}"
> +    basic_create_lookup "${jp_file1}" "${jp_file2}"
> +}
> +
> +test_bad_casefold_lookup () {
> +    bad_basic_create_lookup ${blob_file1} ${blob_file2}
> +}
> +
> +# remove and recreate
> +test_create_and_remove() {
> +    __create_and_remove() {

In fstests, functions and/or variables starting with "_" are reserved
for global names, do_create_and_remove should be fine. And move it out
of the outer function?

> +	basic_create_lookup ${1}
> +	rm -f  ${basedir}/"${1}"
> +	stat   ${basedir}/"${1}" &> /dev/null && \
> +	    _fail "File ${1} was not removed using exact name"
> +
> +	basic_create_lookup ${1}
> +	rm -f  ${basedir}/"${2}"
> +	stat   ${basedir}/"${1}" &> /dev/null && \
> +	    _fail "File ${1} was not removed using inexact name"
> +    }
> +
> +    basedir=${test_casefold}/create_and_remove
> +    mkdir -p ${basedir}
> +
> +    _set_casefold_attr ${basedir}
> +
> +    __create_and_remove "${pt_file1}" "${pt_file2}"
> +    __create_and_remove "${jp_file1}" "${jp_file2}"
> +    __create_and_remove "${ar_file1}" "${ar_file2}"
> +    __create_and_remove "${fr_file1}" "${fr_file2}"
> +
> +}
> +
> +test_casefold_flag_basic () {
> +    basedir=${test_casefold}/basic
> +    mkdir -p ${basedir}
> +
> +    _set_casefold_attr ${basedir}
> +    _is_casefolded_dir ${basedir} || _fail "Casefold attribute wasn't set"
> +
> +    _unset_casefold_attr ${basedir}
> +    _is_casefolded_dir ${basedir} && _fail "Casefold attribute is still set"
> +}
> +
> +test_casefold_flag_removal () {
> +    basedir=${test_casefold}/casefold_flag_removal
> +    mkdir -p ${basedir}
> +
> +    _set_casefold_attr ${basedir}
> +    touch ${basedir}/${filename1}
> +
> +    # Try to remove +F attribute on non empty directory
> +    _try_unset_casefold_attr ${basedir} && \
> +	_fail "Shouldn't be able to remove casefold attribute of non-empty directory"

So here we could just do "_unset_casefold_attr $basename" and log the
expected error message (maybe with some filters to make the output
deterministic) in .out file and we can get rid of the _try functions.

> +}
> +
> +# Test Inheritance of casefold flag
> +test_casefold_flag_inheritance () {
> +    dirpath1="d1/d2/d3"
> +    dirpath2="D1/D2/D3"
> +
> +    basedir=${test_casefold}/flag_inheritance
> +
> +    mkdir -p ${basedir}
> +    _set_casefold_attr ${basedir}
> +
> +    mkdir -p ${basedir}/${dirpath1}
> +
> +    _is_casefolded_dir ${basedir}/${dirpath1} || \
> +	_fail "Casefold attribute should be inherited"
> +    ls  ${basedir}/${dirpath2} &>/dev/null || \
> +	_fail "Dir Lookup failed."
> +    _casefold_check_exact_name "${basedir}" "${dirpath1}" || \
> +	_fail "Created directories wrong name."
> +
> +    touch ${basedir}/${dirpath2}/${filename1}
> +    ls  ${basedir}/${dirpath1}/${filename2} &>/dev/null || \
> +	_fail "Couldn't create file on casefolded parent."
> +}
> +
> +# Test nesting of sensitive directory inside insensitive directory.
> +test_nesting_sensitive_insensitive_tree_simple() {
> +    basedir=${test_casefold}/sd1
> +    mkdir -p ${basedir}
> +    _set_casefold_attr ${basedir}
> +
> +    mkdir -p ${basedir}/sd1
> +    _set_casefold_attr ${basedir}/sd1
> +
> +    mkdir ${basedir}/sd1/sd2
> +    _unset_casefold_attr ${basedir}/sd1/sd2
> +
> +    touch ${basedir}/sd1/sd2/${filename1}
> +    ls ${basedir}/sd1/sd2/${filename1} &>/dev/null || \
> +	_fail "Exact nested file lookup failed."
> +    ls ${basedir}/sd1/SD2/${filename1} &>/dev/null || \
> +	_fail "Nested file lookup failed."
> +    ls ${basedir}/sd1/SD2/${filename2} &>/dev/null && \
> +	_fail "Wrong file lookup passed, should have fail."
> +}
> +
> +test_nesting_sensitive_insensitive_tree_complex() {
> +    # Test nested-directories
> +    basedir=${test_casefold}/nesting
> +    mkdir -p ${basedir}
> +
> +    _set_casefold_attr ${basedir}
> +
> +    mkdir ${basedir}/nd1
> +    _set_casefold_attr ${basedir}/nd1
> +    mkdir ${basedir}/nd1/nd2
> +    _unset_casefold_attr ${basedir}/nd1/nd2
> +    mkdir ${basedir}/nd1/nd2/nd3
> +    _set_casefold_attr ${basedir}/nd1/nd2/nd3
> +    mkdir ${basedir}/nd1/nd2/nd3/nd4
> +    _unset_casefold_attr ${basedir}/nd1/nd2/nd3/nd4
> +    mkdir ${basedir}/nd1/nd2/nd3/nd4/nd5
> +    _set_casefold_attr ${basedir}/nd1/nd2/nd3/nd4/nd5
> +
> +    ls ${basedir}/ND1/ND2/nd3/ND4/nd5 &>/dev/null || \
> +	_fail "Nest-dir Lookup failed."
> +    ls ${basedir}/nd1/nd2/nd3/nd4/ND5 &>/dev/null && \
> +	_fail "ND5: Nest-dir Lookup passed, it should fail."
> +    ls ${basedir}/nd1/nd2/nd3/ND4/nd5 &>/dev/null || \
> +	_fail "Nest-dir Lookup failed."
> +    ls ${basedir}/nd1/nd2/ND3/nd4/ND5 &>/dev/null && \
> +	_fail "ND3: Nest-dir Lookup passed, it should fail."
> +}
> +
> +test_symlink_with_inexact_name () {
> +    basedir=${test_casefold}/symlink
> +    mkdir -p ${basedir}
> +
> +    _set_casefold_attr ${basedir}
> +
> +    mkdir ${basedir}/ind1
> +    mkdir ${basedir}/ind2
> +    _set_casefold_attr ${basedir}/ind1
> +    touch ${basedir}/ind1/target
> +    ln -s ${basedir}/ind1/TARGET ${basedir}/ind2/link
> +    [ -L ${basedir}/ind2/link ] || _fail "Not a symlink."
> +    readlink -e ${basedir}/ind2/link &>/dev/null || _fail "Symlink failed"
> +}
> +
> +
> +# Name-preserving tests
> +# We create a file with a name, delete it and create again with an
> +# equivalent name.  If the negative dentry wasn't invalidated, the
> +# file might be created using $1 instead of $2.
> +test_name_preserve () {
> +    __test_name_preserve () {
> +	touch ${basedir}/${1}
> +	rm ${basedir}/${1}
> +
> +	touch ${basedir}/${2}
> +	_casefold_check_exact_name ${basedir} ${2} ||
> +	    _fail "${2} was not created with exact name"
> +    }
> +    basedir=${test_casefold}/"test_name_preserve"
> +    mkdir -p ${basedir}
> +
> +    _set_casefold_attr ${basedir}
> +
> +    __test_name_preserve "${pt_file1}" "${pt_file2}"
> +    __test_name_preserve "${jp_file1}" "${jp_file2}"
> +    __test_name_preserve "${ar_file1}" "${ar_file2}"
> +    __test_name_preserve "${fr_file1}" "${fr_file2}"
> +}
> +
> +test_dir_name_preserve () {
> +
> +    __test_dir_name_preserve () {
> +	mkdir ${basedir}/${1}
> +	rmdir ${basedir}/${1}
> +
> +	mkdir ${basedir}/${2}
> +	_casefold_check_exact_name ${basedir} ${2} ||
> +	    _fail "${2} was not created with exact name"
> +    }
> +
> +    basedir=${test_casefold}/"dir-test_name_preserve"
> +    mkdir -p ${basedir}
> +
> +    _set_casefold_attr ${basedir}
> +
> +    __test_dir_name_preserve "${pt_file1}" "${pt_file2}"
> +    __test_dir_name_preserve "${jp_file1}" "${jp_file2}"
> +    __test_dir_name_preserve "${ar_file1}" "${ar_file2}"
> +    __test_dir_name_preserve "${fr_file1}" "${fr_file2}"
> +}
> +
> +# Name reuse for CI search
> +test_name_reuse () {
> +    basedir=${test_casefold}/reuse
> +
> +    mkdir ${basedir}
> +    _set_casefold_attr ${basedir}
> +
> +    reuse1=fileX
> +    reuse2=FILEX
> +
> +    touch ${basedir}/${reuse1}
> +    rm -f ${basedir}/${reuse1} || _fail "File lookup failed."
> +    touch ${basedir}/${reuse2}
> +    _casefold_check_exact_name "${basedir}" "${reuse2}" || \
> +	_fail "File created with wrong name"
> +    _casefold_check_exact_name "${basedir}" "${reuse1}" && \
> +	_fail "File created with the old name"
> +}
> +
> +# filepath same name
> +test_create_with_same_name () {
> +    basedir=${test_casefold}/same_name
> +    mkdir ${basedir}
> +
> +    _set_casefold_attr ${basedir}
> +
> +    mkdir -p ${basedir}/same1/same1
> +    touch ${basedir}/SAME1/sAME1/sAMe1
> +    touch -c ${basedir}/SAME1/sAME1/same1 ||
> +	_fail "Trying to create a new file instead of using old one"
> +}
> +
> +# file Rename
> +test_file_rename () {
> +    basedir=${test_casefold}/rename
> +    mkdir -p ${basedir}
> +    _set_casefold_attr ${basedir}
> +
> +    mv ${basedir}/rename ${basedir}/rename &>/dev/null && \
> +	_fail "mv to an equivalent name shouldn't work"
> +    _casefold_check_exact_name ${basedir} "rename"  || \
> +	_fail "Name shouldn't change."
> +}
> +
> +test_casefold_openfd () {
> +    # Test openfd with casefold.
> +    # 1. Delete a file after gettings its fd.
> +    # 2. Then create new dir with same name
> +    ofd1="openfd"
> +    ofd2="OPENFD"
> +
> +    basedir=${test_casefold}/openfd
> +    mkdir -p ${basedir}
> +    _set_casefold_attr ${basedir}
> +
> +    exec 3<> ${basedir}/${ofd1}
> +    rm -rf ${basedir}/${ofd1}
> +    mkdir ${basedir}/${ofd2}
> +    [ -d ${basedir}/${ofd2} ]   || _fail "Not a directory"
> +    _casefold_check_exact_name ${basedir} "${ofd2}" ||
> +	_fail "openfd file was created using old name"
> +    rm -rf ${basedir}/${ofd2}
> +    exec 3>&-
> +}
> +
> +test_casefold_openfd2 () {
> +    # Test openfd with casefold.
> +    # 1. Delete a file after gettings its fd.
> +    # 2. Then create new file with same name
> +    # 3. Read from open-fd and write into new file.
> +
> +    basedir=${test_casefold}/openfd2
> +    mkdir ${basedir}
> +    _set_casefold_attr ${basedir}
> +
> +    date > ${basedir}/${ofd1}
> +    exec 3<> ${basedir}/${ofd1}
> +    rm -rf ${basedir}/${ofd1}
> +    touch ${basedir}/${ofd1}
> +    [ -f ${basedir}/${ofd2} ] || _fail "Not a file"
> +    read data <&3
> +    echo $data >> ${basedir}/${ofd1}
> +    exec 3>&-
> +}
> +
> +test_hard_link_lookups () {
> +    # Create hardlink for casefold dir file and inside regular dir.
> +    basedir=${test_casefold}/hard_link
> +    mkdir ${basedir}
> +    _set_casefold_attr ${basedir}
> +
> +    touch ${basedir}/h1
> +    ln ${basedir}/H1 ${SCRATCH_MNT}/h1
> +    cnt=`stat -c %h ${basedir}/h1`
> +    [ $cnt -eq 1 ] && _fail "Unable to create hardlink"
> +
> +    # Create hardlink for casefold dir file and inside regular dir.
> +    touch ${SCRATCH_MNT}/h2
> +    ln ${SCRATCH_MNT}/h2 ${basedir}/H2
> +    cnt=`stat -c %h ${basedir}/h2`
> +    [ $cnt -eq 1 ] && _fail "Unable to create hardlink"
> +}
> +
> +test_xattrs_lookups () {
> +    basedir=${test_casefold}/xattrs
> +    mkdir ${basedir}
> +    _set_casefold_attr ${basedir}
> +
> +    mkdir -p ${basedir}/x
> +    setfattr -n user.foo -v bar ${basedir}/x
> +    getfattr -n user.foo ${basedir}/x &>/dev/null || \

Use $SETFATTR_PROG and $GETFATTR_PROG and _require_attrs in the
beginning.

> +	_fail "Casefold xattr lookup failed."
> +
> +    touch ${basedir}/x/f1
> +    setfattr -n user.foo -v bar ${basedir}/x/f1
> +    getfattr -n user.foo ${basedir}/x/f1 &>/dev/null || \
> +	_fail "Casefold xattr lookup failed."
> +}
> +
> +test_lookup_large_directory() {
> +    # Perform Casefold lookup on large dir
> +
> +    basedir=${test_casefold}/large
> +    mkdir ${basedir}
> +    _set_casefold_attr ${basedir}
> +
> +    touch $(seq -f "${basedir}/file%g" 0 2000)
> +    stat  $(seq -f "${basedir}/FILE%g" 0 2000) &>/dev/null || \
> +	_fail "Case on large dir failed"
> +}
> +
> +test_strict_mode_invalid_filename () {
> +    __strict_mode_invalid_filename () {
> +	touch  "${basedir}/${1}" &>/dev/null && \
> +	    _fail "Invalid UTF-8 sequence ${1} should fail at creation time."
> +    }
> +
> +    __strict_mode_invalid_filename ${blob1}
> +    __strict_mode_invalid_filename ${blob2}
> +}
> +
> +#############
> +# Run tests #
> +#############
> +
> +_scratch_mkfs_casefold 2>&1 1>/dev/null
> +
> +_scratch_mount
> +
> +_check_dmesg_for \
> +    "\(${sdev}\): Using encoding defined by superblock: utf8" || \
> +    _fail "Could not mount with encoding: utf8"
> +
> +test_casefold_flag_basic
> +test_casefold_lookup
> +test_bad_casefold_lookup
> +test_create_and_remove
> +test_casefold_flag_removal
> +test_casefold_flag_inheritance
> +test_nesting_sensitive_insensitive_tree_simple
> +test_nesting_sensitive_insensitive_tree_complex
> +test_symlink_with_inexact_name
> +test_name_preserve
> +test_dir_name_preserve
> +test_name_reuse
> +test_create_with_same_name
> +test_file_rename
> +test_casefold_openfd
> +test_casefold_openfd2
> +test_hard_link_lookups
> +test_xattrs_lookups
> +test_lookup_large_directory
> +
> +_scratch_unmount 2>&1

No need to do redirection, test harness captures both stdout and stderr.

> +_check_scratch_fs
> +
> +# Test Strict Mode
> +_scratch_mkfs_casefold_strict 2>&1 1>/dev/null

Better to redirect mkfs output to $seqres.full for debug purpose, e.g.

_scratch_mkfs_casefold_strict >>$seqres.full 2>&1

> +_scratch_mount 2>&1

No need to do redirection here either.

> +
> +test_strict_mode_invalid_filename
> +
> +_scratch_unmount 2>&1

Same here.

> +_check_scratch_fs

And if we do fsck after each test phase, we could replace
_require_scratch with _require_scratch_nocheck to let harness know
there's no need to do fsck again after test.

Thanks,
Eryu

> +
> +echo "Silence is golden"
> +status=0
> +exit
> diff --git a/tests/ext4/036.out b/tests/ext4/036.out
> new file mode 100644
> index 000000000000..ed460d98f87c
> --- /dev/null
> +++ b/tests/ext4/036.out
> @@ -0,0 +1,2 @@
> +QA output created by 036
> +Silence is golden
> diff --git a/tests/ext4/group b/tests/ext4/group
> index d27ec893333b..82639c13f8bb 100644
> --- a/tests/ext4/group
> +++ b/tests/ext4/group
> @@ -38,6 +38,7 @@
>  033 auto ioctl resize
>  034 auto quick quota
>  035 auto quick resize
> +036 auto quick casefold
>  271 auto rw quick
>  301 aio auto ioctl rw stress defrag
>  302 aio auto ioctl rw stress defrag
> -- 
> 2.20.1
>