[U-Boot] [RFC 1/3] test/py: convert fs-test.sh to pytest

[AKASHI Takahiro <takahiro.akashi@linaro.org>]

Hi Tuomas,

Thank you for interesting pointers.

On Thu, Aug 30, 2018 at 01:56:41PM +0300, Tuomas Tynkkynen wrote:
> Hi Heinrich, Takahiro
> 
> On 08/30/2018 01:26 PM, AKASHI Takahiro wrote:
> >On Thu, Aug 30, 2018 at 12:01:32PM +0200, Heinrich Schuchardt wrote:
> >>On 08/30/2018 08:52 AM, AKASHI Takahiro wrote:
> >>>On Wed, Aug 29, 2018 at 11:36:51PM +0200, Heinrich Schuchardt wrote:
> >>>>On 08/23/2018 09:25 AM, AKASHI Takahiro wrote:
> >>>>>In this commit, the same set of test cases as in test/fs/fs-test.sh
> >>>>>is provided using pytest framework.
> >>>>>Actually, fs-test.sh provides three variants:"sb" (sb command), "nonfs"
> >>>>>(fatxx and etc.) and "fs" (hostfs), and this patch currently supports
> >>>>>only "nonfs" variant; So it is not a replacement of fs-test.sh for now.
> >>>>>
> >>>>>Simple usage:
> >>>>>   $ py.test test/py/tests/test_fs [<other options>]
> >>>>>
> >>>>>You may also specify filesystem types to be tested:
> >>>>>   $ py.test test/py/tests/test_fs --fs-type fat32 [<other options>]
> >>>>>
> >>>>>Signed-off-by: AKASHI Takahiro <takahiro.akashi@linaro.org>
> >>>>>---
> >>>>>  test/py/tests/test_fs/conftest.py    | 175 +++++++++++++++++++
> >>>>>  test/py/tests/test_fs/fstest_defs.py |  10 ++
> >>>>>  test/py/tests/test_fs/test_basic.py  | 246 +++++++++++++++++++++++++++
> >>>>>  3 files changed, 431 insertions(+)
> >>>>>  create mode 100644 test/py/tests/test_fs/conftest.py
> >>>>>  create mode 100644 test/py/tests/test_fs/fstest_defs.py
> >>>>>  create mode 100644 test/py/tests/test_fs/test_basic.py
> >>>>>
> >>>>>diff --git a/test/py/tests/test_fs/conftest.py b/test/py/tests/test_fs/conftest.py
> >>>>>new file mode 100644
> >>>>>index 000000000000..fefeb4c9663f
> >>>>>--- /dev/null
> >>>>>+++ b/test/py/tests/test_fs/conftest.py
> >>>>>@@ -0,0 +1,175 @@
> >>>>>+# SPDX-License-Identifier:      GPL-2.0+
> >>>>>+# Copyright (c) 2018, Linaro Limited
> >>>>>+# Author: Takahiro Akashi <takahiro.akashi@linaro.org>
> >>>>>+
> >>>>>+import pytest
> >>>>>+import re
> >>>>>+from subprocess import call, check_call, check_output, CalledProcessError
> >>>>>+from fstest_defs import *
> >>>>>+
> >>>>>+supported_fs_basic = ['fat16', 'fat32', 'ext4']
> >>>>>+
> >>>>>+#
> >>>>>+# Filesystem test specific setup
> >>>>>+#
> >>>>>+def pytest_addoption(parser):
> >>>>>+    parser.addoption('--fs-type', action='append', default=None,
> >>>>>+        help='Targeting Filesystem Types')
> >>>>>+
> >>>>>+def pytest_configure(config):
> >>>>>+    global supported_fs_basic
> >>>>>+
> >>>>>+    def intersect(listA, listB):
> >>>>>+        return  [x for x in listA if x in listB]
> >>>>>+
> >>>>>+    supported_fs = config.getoption('fs_type')
> >>>>>+    if supported_fs:
> >>>>>+        print("*** FS TYPE modified: %s" % supported_fs)
> >>>>>+        supported_fs_basic =  intersect(supported_fs, supported_fs_basic)
> >>>>>+
> >>>>>+def pytest_generate_tests(metafunc):
> >>>>>+    if 'fs_obj_basic' in metafunc.fixturenames:
> >>>>>+        metafunc.parametrize('fs_obj_basic', supported_fs_basic,
> >>>>>+            indirect=True, scope='module')
> >>>>>+
> >>>>>+#
> >>>>>+# Helper functions
> >>>>>+#
> >>>>>+def fstype_to_ubname(fs_type):
> >>>>>+    if re.match('fat', fs_type):
> >>>>>+        return 'fat'
> >>>>>+    else:
> >>>>>+        return fs_type
> >>>>>+
> >>>>>+def check_ubconfig(config, fs_type):
> >>>>>+    if not config.buildconfig.get('config_cmd_%s' % fs_type, None):
> >>>>>+        pytest.skip('.config feature "CMD_%s" not enabled' % fs_type.upper())
> >>>>>+    if not config.buildconfig.get('config_%s_write' % fs_type, None):
> >>>>>+        pytest.skip('.config feature "%s_WRITE" not enabled'
> >>>>>+        % fs_type.upper())
> >>>>>+
> >>>>>+def mk_fs(config, fs_type, size, id):
> >>>>>+    fs_img = '%s.%s.img' % (id, fs_type)
> >>>>>+    fs_img = config.persistent_data_dir + '/' + fs_img
> >>>>>+
> >>>>>+    if fs_type == 'fat16':
> >>>>>+        mkfs_opt = '-F 16'
> >>>>>+    elif fs_type == 'fat32':
> >>>>>+        mkfs_opt = '-F 32'
> >>>>>+    else:
> >>>>>+        mkfs_opt = ''
> >>>>>+
> >>>>>+    if re.match('fat', fs_type):
> >>>>>+        fs_lnxtype = 'vfat'
> >>>>>+    else:
> >>>>>+        fs_lnxtype = fs_type
> >>>>>+
> >>>>>+    count = (size + 1023) / 1024
> >>>>>+
> >>>>>+    try:
> >>>>>+        check_call('rm -f %s' % fs_img, shell=True)
> >>>>>+        check_call('dd if=/dev/zero of=%s bs=1K count=%d'
> >>>>>+            % (fs_img, count), shell=True)
> >>>>>+        check_call('mkfs.%s %s %s'
> >>>>>+            % (fs_lnxtype, mkfs_opt, fs_img), shell=True)
> >>>>>+        return fs_img
> >>>>>+    except CalledProcessError:
> >>>>>+        call('rm -f %s' % fs_img, shell=True)
> >>>>>+        raise
> >>>>>+
> >>>>>+#
> >>>>>+# Fixture for basic fs test
> >>>>>+#     derived from test/fs/fs-test.sh
> >>>>>+#
> >>>>>+# NOTE: yield_fixture was deprecated since pytest-3.0
> >>>>>+ at pytest.yield_fixture()
> >>>>>+def fs_obj_basic(request, u_boot_config):
> >>>>>+    fs_type = request.param
> >>>>>+    fs_img = ''
> >>>>>+
> >>>>>+    fs_ubtype = fstype_to_ubname(fs_type)
> >>>>>+    check_ubconfig(u_boot_config, fs_ubtype)
> >>>>>+
> >>>>>+    mount_dir = u_boot_config.persistent_data_dir + '/mnt'
> >>>>>+    small_file = mount_dir + '/' + SMALL_FILE
> >>>>>+    big_file = mount_dir + '/' + BIG_FILE
> >>>>>+    try:
> >>>>>+
> >>>>>+        # 3GiB volume
> >>>>>+        fs_img = mk_fs(u_boot_config, fs_type, 0xc0000000, '3GB')
> >>>>>+
> >>>>>+        # Mount the image so we can populate it.
> >>>>>+        check_call('mkdir -p %s' % mount_dir, shell=True)
> >>>>>+        check_call('sudo mount -o loop,rw %s %s'
> >>>>>+            % (fs_img, mount_dir), shell=True)
> >>>>
> >>>>Should I grant sudo to anybody who can commit to U-Boot?
> >>>
> >>>I don't get your point.
> >>>I think using "sudo" solely in testing should be allowed.
> >>>
> >>>>Just use exfat-fuse and fuse2fs.
> >>>
> >>>It will not be a good idea to use those tools which are not
> >>>provided in every distribution.
> >>>I'd like to leave "sudo mount" statement as a backstop.
> >>
> >>Fuse is a base functionality of Linux. On Travis we are using the Ubuntu
> >>distribution which contains said fuse file systems. Which distribuition
> >>does not have it?
> >>
> >>Using sudo for me is a NO-NO. I will not run any test that uses sudo.
> >
> >So this means that you have never tested file system using test-fs.sh.
> >
> >Since my script is logically "general", it can, if we want, run against
> >other file systems as well. "sudo mount" is the only solution for those cases.
> >
> There are two general non-root implementations that I know of:
> 
> 1) http://libguestfs.org/ which IIRC launches a small Linux VM in QEMU
>    to do the filesystem accesses. I am not sure if the performance would be
>    acceptable without KVM (which I assume we don't have in Travis).

I didn't dig into this tool, but if it is all about VM on qemu,
the discussion here, whether we should be allowed to use sudo or not,
would be pointless as we can do whatever we want to do under VM.

> 2) https://github.com/lkl/linux which is a port of Linux to run in userspace
>    as a library. It comes with tools like cptofs and lklfuse to access any
>    filesystem Linux has a driver for.

It appears to be a kind of libos or unikernel, or rather,
resembles a sandbox of u-boot?

> Sadly lkl isn't packaged in many distros.

Too bad.

-Takahiro AKASHI

> - Tuomas