ext4 non-kernel standalone dev version for testing

ext4 non-kernel standalone dev version for testing

[U.Mutlu]

Hi,
can the developer-version (ie. that with the crypto extension) be tested 
outside the kernel?
Ie. testing with file type filesystem, ie. file.img, on a system with old kernel.

Thx

Re: ext4 non-kernel standalone dev version for testing

[Theodore Ts'o]

On Fri, May 01, 2015 at 03:29:11AM +0200, U.Mutlu wrote:
> can the developer-version (ie. that with the crypto extension) be tested
> outside the kernel?
> Ie. testing with file type filesystem, ie. file.img, on a system with old kernel.

No, but you can test under KVM.  This is how I normally do most of my
ext4 development.  Things are a little complicated at the moment
because some e2fsprogs changes are needed that aren't in the binaries
in the kvm-xfstests image, but I'll try to get a new version sent out
soon.

						- Ted


Quick start instructions for kvm-xfstests
==========================================

0.  Make sure qemu with kvm support is installed on your system.

1.  Run the following commands:

    git clone git://git.kernel.org/pub/scm/fs/ext2/xfstests-bld.git fstests
    cd fstests/kvm-xfstests
    wget -O test_appliance/root_fs.img https://www.kernel.org/pub/linux/kernel/people/tytso/kvm-xfstests/root_fs.img.i386

Substitute "i386" for "x86_64" in the above URL if you want to test
using a native 64-bit userspace.  It should be possible to use the
32-bit userspace with a 64-bit kernel, although you may run into some
rough spots via-a-vis compat ioctls working correctly.  (If you do,
let me know; those are bugs that should be fixed.  :-)

2.  In the fstests/kvm-xfstests directory, take a look at the "config"
    file and either edit that file in place, or put override values in
    ~/.config/kvm-xfstests or config.custom.  The latter is preferred.

3.  Build a kernel with all of the necessary drives for kvm built into
    the kernel (i.e., no modules!).  There are some sample kernel
    configs in the kernel-configs directory.

4.  Run "./kvm-xfstests smoke" to do a quick test.  Or "./kvm-xfstests
    -g auto" to do a full test.  You can also run specific tests on
    specific configurations, i.e., "./kvm-xfstests -c bigalloc
    generic/013 generic/127".   To run a shell, use "./kvm-xfstests shell"

For more information, please see:

https://git.kernel.org/cgit/fs/ext2/xfstests-bld.git/tree/kvm-xfstests/README


Building a 32-bit kernel
========================


To build a 32-bit kernel, create a /build/ext4 directory (or change
directory name as you wish) and place a kernel configuration file in
/build/ext4/.config.  (There are some example config files in the
kernel-configs directory of the xfstests-bld git repo).  Then run:

make O=/build/ext4 ARCH=i386 -j16 $*

The reason for using a separate build directory is you can maintain a
separate build directory for 64-bit builds, which you can now build
using:

make O=/build/ext4-64 ARCH=x86_64 -j16 $*

I generally create shell scripts which look like this:

#!/bin/dash
cd /usr/projects/linux/ext4
make O=/build/ext4-64 ARCH=x86_64 -j16 $*

And store them as "make-ext4", "make-ext4-64", "make-ext4-crypto",
etc., in /usr/projects/linux.  The reason for having the "cd" command
is so that if you accidentally run "../make-ext4" while you are in the
/usr/projects/linux/ext4-crypto directory, instead of the
/usr/projects/linux/ext4 directory (for example), you won't end up
accidentally contaminating the build directory.

The remote ports feature
========================

While kvm-xfstests is running, you can telnet to a number of ports
(which are bound to localhost).  Ports 7500, 7501, and 7502 will
connect you to a shell prompts while the tests are running (if you
want to check on /proc/slabinfo, enable tracing, etc.)  You can also
use these ports in conjuction with "kvm-xfstests shell" if you want
additional windows to capture traces using ftrace.

You can also access the qemu monitor on port 7498, and you debug the
kernel using remote gdb on localhost port 7499.  Just run "gdb
/path/to/vmlinux", and then use the command "target remote
localhost:7499".  (Pro tip: it's helpful to temporarily add
"EXTRA_CFLAGS += -O0" to fs/{ext4,jbd2}/Makefile, and to make sure you
have CONFIG_DEBUG_INFO, CONFIG_DEBUG_INFO_DWARF4, and
CONFIG_FRAME_POINTER enabled.)


Miscellaneous kernel hacking hints
==================================

The following hints may be useful to folks who are new to kernel
development.

Using ccache
------------

Enabling ccache can really help speed up your kernel builds.  I
strongly recommend it.

Examples of using ftrace
------------------------

This is an example of how to debug the lazytime feature (which is
when you mount a file system using -m lazytime using a 4.0-rcX and
later kernels).

cd /sys/kernel/debug/tracing
echo 1 > events/writeback/writeback_lazytime/enable
echo 1 > events/writeback/writeback_lazytime_iput/enable
echo "state & 2048" > events/writeback/writeback_dirty_inode_enqueue/filter
echo 1 > events/writeback/writeback_dirty_inode_enqueue/enable
echo 1 > events/ext4/ext4_other_inode_update_time/enable
cat trace_pipe

The definition of the tracepoints can be found in
include/linux/trace/events.  The tracepoints used by ext4 and be found
in ext4.h, and by jbd2 in the jbd2.h files in that directory, and so
on.

For more information, please see:

http://lwn.net/Articles/365835/
http://lwn.net/Articles/366796/
http://lwn.net/Articles/370423/

Re: ext4 non-kernel standalone dev version for testing

[U.Mutlu]

Theodore Ts'o wrote on 05/01/2015 04:13 PM:
> On Fri, May 01, 2015 at 03:29:11AM +0200, U.Mutlu wrote:
>> can the developer-version (ie. that with the crypto extension) be tested
>> outside the kernel?
>> Ie. testing with file type filesystem, ie. file.img, on a system with old kernel.
>
> No, but you can test under KVM.  This is how I normally do most of my
> ext4 development.  Things are a little complicated at the moment
> because some e2fsprogs changes are needed that aren't in the binaries
> in the kvm-xfstests image, but I'll try to get a new version sent out
> soon.
>
> 						- Ted
>
>
> Quick start instructions for kvm-xfstests
...


Thx, but unfortunately the Makefile reports an error:

...
checking for attr_list_by_handle in -lhandle... yes
checking attr/xattr.h usability... no
checking attr/xattr.h presence... no
checking for attr/xattr.h... no

FATAL ERROR: attr/xattr.h does not exist.
Install the extended attributes (attr) development package.
Alternatively, run "make install-dev" from the attr source.
Makefile:83: recipe for target 'include/builddefs' failed
make[1]: *** [include/builddefs] Error 1
make[1]: Leaving directory '/root/ext4crypto-test/fstests/xfstests-dev'


root@c22:/root/ext4crypto-test/fstests/attr# find . -iname "*xattr*" -ls
35786444    4 -rw-r--r--   1 root     root         2297 May  1 18:52 
./include/xattr.h


root@c22:/root/ext4crypto-test/fstests/attr/include# ls -l
total 60
lrwxrwxrwx 1 root root    1 May  1 19:46 attr -> .
-rw-r--r-- 1 root root 7232 May  1 18:52 attributes.h
-rw-r--r-- 1 root root 2180 May  1 19:46 builddefs
-rw-r--r-- 1 root root 2155 May  1 18:52 builddefs.in
-rw-r--r-- 1 root root 4633 May  1 18:52 buildmacros
-rw-r--r-- 1 root root 1728 May  1 18:52 buildrules
-rw-r--r-- 1 root root 1344 May  1 19:40 config.h
-rw-r--r-- 1 root root 1282 May  1 18:52 config.h.in
-rw-r--r-- 1 root root  846 May  1 18:52 error_context.h
-rw-r--r-- 1 root root  688 May  1 18:52 libattr.h
-rw-r--r-- 1 root root  521 May  1 18:52 Makefile
-rw-r--r-- 1 root root  340 May  1 18:52 misc.h
-rw-r--r-- 1 root root 1272 May  1 18:52 walk_tree.h
-rw-r--r-- 1 root root 2297 May  1 18:52 xattr.h


Where/how should I fix this xattr.h build-bug?

System is debian 8 x86_64, testing in a chrooted environment (lxc app container).

cu
Uenal

Re: ext4 non-kernel standalone dev version for testing

[Theodore Ts'o]

On Fri, May 01, 2015 at 08:31:22PM +0200, U.Mutlu wrote:
> 
> Thx, but unfortunately the Makefile reports an error:

First of all, it's not necessary for you to build the whole setup if
you just want to use kvm-xfstests.  Just download the root_fs.img file
as described in the quickstart.  In fact, if you follow the quickstart
documentation I sent you **exactly** (which you will note doesn't
include trying to build things), it should be enough so you can at
least get something set up for testing generic ext4.  I will create an
updated root_fs.img that can be used for testing encryption in a few
days.


That being said, the build *should* have worked, but you need to send
me the full output of the make output if you want me to help you debug
it.  (i.e., run "make clean ; make >& MAKELOG", and then send me the
output of MAKELOGf.)  It would also be useful to send me the output of
this command line, so I can see what versions of the subcomponents you
are using:

for i in $(find . -name .git) ; do echo $i $(cd $i ; git describe --always); done

specifically, the quota tree recently got autoconfiscated, and so at
least at one point it wasn't building smoothly; doing a "(cd quota ;
git checkout 022aabf)" is a good idea.  Similarly the fio tree is not
necessarily always stable; it's sometimes a good idea to check out the
most recent release tag, instead of using the bleeding edge
development branch for safety reasons.  Here are the versions that I'm
currently personally using for the various subcomponents needed to
build xfstests:

fio	   	 fio-2.2.6 (Fri, 27 Feb 2015 08:30:04 -0700)
quota		 	   022aabf (Wed, 26 Nov 2014 10:22:08 +0100)
xfsprogs		   v3.2.2 (Thu, 4 Dec 2014 07:56:44 +1100)
xfstests-bld		   9fadb2d (Mon, 6 Apr 2015 18:38:24 -0400)
xfstests		   linux-v3.8-640-gfacff60 (Thu, 2 Apr 2015 09:41:15 +1100)


Finally, note that when the build is done, there are still a few more
steps before you can generate a root_fs.img that can be used with kvm,
and that it requires that you are using a Debian or Ubuntu system (and
I only really test on Debian).  The specifics of what you need to do
can be found in the do-all script if you really want to go down this
path ---- but I emphasize, this is all not really necessary unless you
want to do your own test development and/or improve the kvm-xfstests
test appliance.

If you don't, it's *much* simpler to just follow the quickstart
document I sent you very carefully, and just download the root_fs.img
file using wget as I described in step #1.

Finally, out of curiosity, what is your interest in testing/using the
ext4 encryption feature?  It's still very much early days, and so if I
know what you are trying to do, I might be able to provide more
assistance.  Please note that it requires a pre-release version of
e2fsprogs, and with all new features, it's not at the stage where I
would recommend that users use it for critical data until it has gone
through additional testing (and no doubt, bug fixing).

Best regards,

					- Ted