Fwd: Re: Broken nilfs2 filesystem

[Anton Eliasson <devel-17Olwe7vw2dLC78zk6coLg@public.gmane.org>]

I forgot to send this to the list.

-------- Ursprungligt meddelande --------
Ämne: 	Re: Broken nilfs2 filesystem
Datum: 	Fri, 26 Jul 2013 18:49:40 +0200
Från: 	Anton Eliasson <devel-17Olwe7vw2dLC78zk6coLg@public.gmane.org>
Till: 	Vyacheslav Dubeyko <slava-2lV3ebY47BVBDgjK7y7TUQ@public.gmane.org>



Vyacheslav Dubeyko skrev 2013-07-26 14:37:
> Hi Anton,
>
> Do you ready to try to obtain debug output? I am really waiting your
> readiness because your opportunity to reproduce the issue is very
> important.
>
> I think that it needs to enable such configuration options:
> 1. CONFIG_NILFS2_DEBUG_SHOW_ERRORS
> 2. CONFIG_NILFS2_DEBUG_BASE_OPERATIONS
> 3. CONFIG_NILFS2_DEBUG_MDT_FILES
> 4. CONFIG_NILFS2_DEBUG_SEGMENTS_SUBSYSTEM
> 5. CONFIG_NILFS2_DEBUG_BLOCK_MAPPING
> 6. CONFIG_NILFS2_DEBUG_DUMP_STACK
>
> Thanks,
> Vyacheslav Dubeyko.

Hi,
Thanks for the reminder! I tried it out a few weeks ago. My plan then
was to restore the old root system (running Linux 3.9) and /home to the
mechanical hard drive, boot them up and use that system to build the
newer kernel with your kernel. This would make minimal changes to the
system I had experienced the bugs with, I thought.

Building the kernel took me some time but I eventually succeded. I
forgot those configuration options though so I suppose that build did
nothing out of the ordinary, even though it had the patches. I got it to
boot but then ran in to the issue of incompatible video drivers. I've
been down that road before and I did not feel like going back.

My next plan was to wait until 3.10 got released into the core
repository which happened yesterday or so. I would then use my fully
updated system (currently running the 3.10 kernel on an ext4 root
filesystem) to build a kernel with the debug patches. It would be a
slightly different system with many updated packages, but at least the
root filesystem will be a healthy ext4. So that's what I did today.
Buildlogs excerpts are at the bottom of this message.

I got the kernel to boot and X to start. But as soon as anything tried
to read from /home (even just logging into a virtual terminal as a
regular user), that terminal froze. Logging in as root to a different VT
showed that syslog-ng and nilfs_cleanerd took 100% CPU each. After a
while the entire system froze. Pressing Alt+SysRq+R caused the kernel to
spew out an endless stream of call traces to the terminal, leaving no
other way to reboot than a hard reset. After a reboot I found that the
running nilfs_cleanerd does not respond to either a SIGTERM or a SIGKILL.

Changing the mount options of /home from rw,noatime,discard to
ro,norecovery,noatime,discard seems to work. No crashes during login to
a VT. Read-only is no fun though.

Next attempt is rw,nogc,noatime,discard. It seems to work also, though
there are a lot of call traces in dmesg. After just a few reboots and a
few minutes of uptime, /var/log/kernel.log, messages.log and
everything.log have grown to 3.5 GB each. My / is only 30 GB so I can't
sustain this for very long.

I have aborted the experiments for today. kernel.log has 35 million
lines and compresses to 220 MB. I've uploaded it here
(http://antoneliasson.se/publicdump/kernel.log.20130726.gz). What should
I do next?

Build logs
=======

* Download kernel package build scripts and patches (from the Arch Build
System). Unpack and apply downstream patches:

     $ makepkg -o
     ==> Skapar paket: linux 3.10.2-1 (fre jul 26 17:08:16 CEST 2013)
     ==> Retrieving sources...
       -> Hittade linux-3.10.tar.xz
       -> Laddar ner patch-3.10.2.xz...
       % Total    % Received % Xferd  Average Speed   Time Time
Time  Current
                                      Dload  Upload   Total Spent
Left  Speed
     100   178  100   178    0     0    188      0 --:--:-- --:--:--
--:--:--   188
     100 27876  100 27876    0     0  13761      0  0:00:02  0:00:02
--:--:-- 37978
       -> Hittade config
       -> Hittade config.x86_64
       -> Hittade linux.preset
       -> Hittade change-default-console-loglevel.patch
     ==> Validerar källfiler med md5sums...
         linux-3.10.tar.xz ... Godkänd
         patch-3.10.2.xz ... Godkänd
         config ... Godkänd
         config.x86_64 ... Godkänd
         linux.preset ... Godkänd
         change-default-console-loglevel.patch ... Godkänd
     ==> Extracting sources...
       -> Extraherar linux-3.10.tar.xz med bsdtar
       -> Extraherar patch-3.10.2.xz med xz
     ==> Startar prepare()...
     patching file Documentation/parisc/registers
     patching file MAINTAINERS
     patching file Makefile
     patching file arch/arm/boot/dts/imx23.dtsi
     patching file arch/arm/boot/dts/imx28.dtsi
     patching file arch/arm/boot/dts/imx6dl.dtsi
     patching file arch/arm/boot/dts/imx6q.dtsi
     patching file arch/arm/include/asm/mmu_context.h
     patching file arch/arm/kernel/perf_event.c
     [...]
     patching file mm/page_alloc.c
     patching file mm/slab.c
     patching file net/ceph/auth_none.c
     patching file kernel/printk.c
     Hunk #1 succeeded at 56 with fuzz 2 (offset -2 lines).
     ==> Källor är redo.
     $ ls -l
     -rw-r--r-- 1 anton anton 2,7K 26 jul 00.06
alsa-firmware-loading-3.8.8.patch
     -rw-r--r-- 1 anton anton  605 26 jul 00.06
change-default-console-loglevel.patch
     -rw-r--r-- 1 anton anton 142K 26 jul 17.13 config
     -rw-r--r-- 1 anton anton 142K 26 jul 00.06 config~
     -rw-r--r-- 1 anton anton 138K 26 jul 17.13 config.x86_64
     -rw-r--r-- 1 anton anton 138K 26 jul 00.06 config.x86_64~
     -rw-r--r-- 1 anton anton  926 26 jul 00.06 linux.install
     -rw-r--r-- 1 anton anton  376 26 jul 00.06 linux.preset
     drwxr-xr-x 2 anton anton 4,0K 26 jul 17.06
nilfs2-debug-output-patch-set-25-06-2013
     -rw-r--r-- 1 anton anton  13K 26 jul 00.06 PKGBUILD
     drwxr-xr-x 3 anton anton 4,0K 26 jul 17.08 src

* Apply nilfs2 debug patches:

     $ cd src/linux-3.10/
     $ for file in
../../nilfs2-debug-output-patch-set-25-06-2013/*.patch; do patch -p1 <
"$file"; done
     patching file fs/nilfs2/Kconfig
     patching file fs/nilfs2/debug.h
     patching file fs/nilfs2/Kconfig
     patching file fs/nilfs2/debug.h
     patching file fs/nilfs2/dir.c
     patching file fs/nilfs2/file.c
     patching file fs/nilfs2/inode.c
     patching file fs/nilfs2/ioctl.c
     patching file fs/nilfs2/namei.c
     patching file fs/nilfs2/nilfs.h
     [...]
     patching file fs/nilfs2/segbuf.c
     patching file fs/nilfs2/segment.c
     patching file fs/nilfs2/sufile.c
     patching file fs/nilfs2/super.c
     patching file fs/nilfs2/the_nilfs.c
     patching file fs/nilfs2/Kconfig
     patching file fs/nilfs2/debug.h

* Append the following lines to config (just in case) and config.x86_64
(which I assume I will use):

     CONFIG_NILFS2_DEBUG_SHOW_ERRORS=y
     CONFIG_NILFS2_DEBUG_BASE_OPERATIONS=y
     CONFIG_NILFS2_DEBUG_MDT_FILES=y
     CONFIG_NILFS2_DEBUG_SEGMENTS_SUBSYSTEM=y
     CONFIG_NILFS2_DEBUG_BLOCK_MAPPING=y
     CONFIG_NILFS2_DEBUG_DUMP_STACK=y

* Build the kernel. There were some interactive configuration options
concerning nilfs2 that I responded yes to:

     $ makepkg -e
     ==> Skapar paket: linux 3.10.2-1 (fre jul 26 17:16:41 CEST 2013)
     ==> Checking runtime dependencies...
     ==> Checking buildtime dependencies...
     ==> VARNING:  Using existing src/ tree
     ==> Startar build()...
       HOSTCC  scripts/basic/fixdep
       HOSTCC  scripts/kconfig/conf.o
       SHIPPED scripts/kconfig/zconf.tab.c
       SHIPPED scripts/kconfig/zconf.lex.c
       SHIPPED scripts/kconfig/zconf.hash.c
       HOSTCC  scripts/kconfig/zconf.tab.o
       HOSTLD  scripts/kconfig/conf
     scripts/kconfig/conf --silentoldconfig Kconfig
     *
     * Restart config...
     *
     *
     * File systems
     *
     [...]
     NILFS2 file system support (NILFS2_FS) [M/n/y/?] m
       NILFS2 debugging (NILFS2_DEBUG) [N/y/?] (NEW) y
         Use pr_debug() instead of printk() (NILFS2_USE_PR_DEBUG)
[N/y/?] (NEW) y
         Show internal errors (NILFS2_DEBUG_SHOW_ERRORS) [N/y/?] (NEW) y
         Enable dump stack output (NILFS2_DEBUG_DUMP_STACK) [N/y/?] (NEW) y
     [...]
     #
     # configuration written to .config
     #
       SYSHDR arch/x86/syscalls/../include/generated/uapi/asm/unistd_32.h
       SYSHDR arch/x86/syscalls/../include/generated/uapi/asm/unistd_64.h
       SYSHDR arch/x86/syscalls/../include/generated/uapi/asm/unistd_x32.h
       SYSTBL arch/x86/syscalls/../include/generated/asm/syscalls_32.h
       SYSHDR arch/x86/syscalls/../include/generated/asm/unistd_32_ia32.h
       SYSHDR arch/x86/syscalls/../include/generated/asm/unistd_64_x32.h
       SYSTBL arch/x86/syscalls/../include/generated/asm/syscalls_64.h
       WRAP    arch/x86/include/generated/asm/clkdev.h
       CHK     include/generated/uapi/linux/version.h
       UPD     include/generated/uapi/linux/version.h
       CHK     include/generated/utsrelease.h
       UPD     include/generated/utsrelease.h
       HOSTCC  arch/x86/tools/relocs_32.o
     [...]
       INSTALL
/home/anton/build/linux/pkg/linux/lib/firmware/edgeport/down3.bin
       INSTALL
/home/anton/build/linux/pkg/linux/lib/firmware/keyspan_pda/keyspan_pda.fw
       INSTALL
/home/anton/build/linux/pkg/linux/lib/firmware/keyspan_pda/xircom_pgs.fw
       INSTALL
/home/anton/build/linux/pkg/linux/lib/firmware/cpia2/stv0672_vp4.bin
       INSTALL /home/anton/build/linux/pkg/linux/lib/firmware/yam/1200.bin
       INSTALL /home/anton/build/linux/pkg/linux/lib/firmware/yam/9600.bin
       DEPMOD  3.10.2-1-ARCH
     ==> Städar upp efter installationen...
       -> Rensar oönskade filer...
       -> Komprimerar man och info sidor...
     ==> Creating package "linux"...
       -> Skapar .PKGINFO fil...
       -> Lägger till install fil...
       -> Generating .MTREE file...
       -> Komprimerar paket...
     ==> Startar package_linux-headers()...
     ==> Städar upp efter installationen...
       -> Rensar oönskade filer...
       -> Komprimerar man och info sidor...
     ==> Creating package "linux-headers"...
       -> Skapar .PKGINFO fil...
       -> Generating .MTREE file...
       -> Komprimerar paket...
     ==> Startar package_linux-docs()...
     ==> Städar upp efter installationen...
       -> Rensar oönskade filer...
       -> Komprimerar man och info sidor...
     ==> Creating package "linux-docs"...
       -> Skapar .PKGINFO fil...
       -> Generating .MTREE file...
       -> Komprimerar paket...
     ==> Leaving fakeroot environment.
     ==> Kompilering klar: linux 3.10.2-1 (fre jul 26 17:46:42 CEST 2013)

* Install over the stock 3.10 kernel, point /home to the old nilfs2
filesystem in /etc/fstab and reboot.

-- 
Best Regards,
Anton Eliasson



--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html