an incident with reiser4

an incident with reiser4

[sergey ivanov]

This night I had an incident with reiser4 at IMAP mailserver (dovecot).
It was /home, mounted with fstab line:
/dev/evms/home /home reiser4 defaults 1 2
/dev/evms/home is an md3 mirror, contains 3 parts 16,0 Gb each, residing 
on 3 different IDE disks.
There was not anything interesting in logs before the incident.
The incident begin with the following messages:
------------------
Mar 15 02:20:24 parkheights kernel: reiser4[maildrop-ldap(7465)]: 
cbk_level_lookup (fs/reiser4/search.c:1033)[vs-3533]:
Mar 15 02:20:24 parkheights kernel: WARNING: Keys are inconsistent. Fsck?
Mar 15 02:20:24 parkheights kernel: reiser4[maildrop-ldap(7465)]: 
cbk_level_lookup (fs/reiser4/search.c:1033)[vs-3533]:
Mar 15 02:20:24 parkheights kernel: WARNING: Keys are inconsistent. Fsck?
Mar 15 02:20:24 parkheights kernel: reiser4[maildrop-ldap(7465)]: 
cbk_level_lookup (fs/reiser4/search.c:1033)[vs-3533]:
Mar 15 02:20:24 parkheights kernel: WARNING: Keys are inconsistent. Fsck?
Mar 15 02:20:24 parkheights kernel: reiser4[maildrop-ldap(7465)]: 
cut_tree_object (fs/reiser4/tree.c:1747)[nikita-2861]:
Mar 15 02:20:24 parkheights kernel: WARNING: failure: -5
Mar 15 02:20:24 parkheights kernel: reiser4[maildrop-ldap(7465)]: 
delete_inode_common (fs/reiser4/plugin/object.c:1003)[
vs-1216]:
Mar 15 02:20:24 parkheights kernel: WARNING: Failed to delete file body 
1279844
Mar 15 02:20:24 parkheights kernel: reiser4[maildrop-ldap(7465)]: 
cbk_level_lookup (fs/reiser4/search.c:1033)[vs-3533]:
Mar 15 02:20:24 parkheights kernel: WARNING: Keys are inconsistent. Fsck?
Mar 15 02:20:24 parkheights kernel: reiser4[maildrop-ldap(7465)]: 
cut_tree_object (fs/reiser4/tree.c:1747)[nikita-2861]:
Mar 15 02:20:24 parkheights kernel: WARNING: failure: -5
Mar 15 02:20:36 parkheights kernel: reiser4[maildrop-ldap(7480)]: 
cbk_level_lookup (fs/reiser4/search.c:1033)[vs-3533]:
Mar 15 02:20:36 parkheights kernel: WARNING: Keys are inconsistent. Fsck?
Mar 15 02:20:36 parkheights kernel: reiser4[maildrop-ldap(7480)]: 
cbk_level_lookup (fs/reiser4/search.c:1033)[vs-3533]:
Mar 15 02:20:36 parkheights kernel: WARNING: Keys are inconsistent. Fsck?
Mar 15 02:20:36 parkheights kernel: reiser4[maildrop-ldap(7480)]: 
cbk_level_lookup (fs/reiser4/search.c:1033)[vs-3533]:
Mar 15 02:20:36 parkheights kernel: WARNING: Keys are inconsistent. Fsck?
Mar 15 02:20:36 parkheights kernel: reiser4[maildrop-ldap(7480)]: 
cut_tree_object (fs/reiser4/tree.c:1747)[nikita-2861]:
Mar 15 02:20:36 parkheights kernel: WARNING: failure: -5
Mar 15 02:20:36 parkheights kernel: reiser4[maildrop-ldap(7480)]: 
delete_inode_common (fs/reiser4/plugin/object.c:1003)[
vs-1216]:
------------------
And so on,  5603 lines till 06:14 AM.
Then I came to the server and saw the problem. I switched to runlevel 1, 
tried to unmount /home, and waited about 10 minutes, the system looks 
hanged up, without reaction to the Ctrl+Alt+Del.
After 10 minutes I have reset the system, booted to runlevel 1, and 
tried fsck.reiser4 /dev/evms/home. Got answer that the filesystem has 
major problems and I should use --build-fs. Sorry, I have not recorded 
this message. Then I run
# fsck.reiser4 --build-fs /dev/evms/home |tee /root/reiser4.log, and 
after some minutes the filesystem is consystent.
But alas, all the messages was not directed to stdout, but stderr, so 
the file /root/reiser4.log was empty.
Now I have mounted filesystem, and all looks good. But I'm not sure if I 
had lost some data.

I'd like to ask questions:
1) Do people using reiser4 experience such incidents, and if do, how 
often? (I personally use reiser4fs about half a year and all the 
problems I have encountered were of my own mistakes)
2) Is it possible to check information at this filesystem, by reiser4 
tools or some another way, - or I should wait if some of mail users 
complain about corrupted or missing information?
3) Is it possible to identify the problem, or it is gone since I have 
not redirected output of fsck.reiser4 right way?

   With best regards,
       Sergey Ivanov

Re: an incident with reiser4

[Christian]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

sergey ivanov wrote:
> # fsck.reiser4 --build-fs /dev/evms/home |tee /root/reiser4.log, and
> after some minutes the filesystem is consystent.
> But alas, all the messages was not directed to stdout, but stderr, so
> the file /root/reiser4.log was empty.

use "fsck.reiser4 --build-fs /dev/evms/home 2>&1 |tee /root/reiser4.log"
to also get stderr messages to the logfile.

> 2) Is it possible to check information at this filesystem, by reiser4
> tools or some another way, - or I should wait if some of mail users
> complain about corrupted or missing information?

as several other readers here pointed out - reiser4 is still beta, don't
use it in production but use it everwhere else to shake out all the
remaining bugs ;)

if you're already using it in production and fsck.reiser4 reports no more
errors, i can't think of any other way than to diff your current reiserfs
with a snapshot made earlier. but since files under /home and especially
mails are always "in flight", this would require a lot of manual "this
file is different, because..." actions.

> 3) Is it possible to identify the problem, or it is gone since I have
> not redirected output of fsck.reiser4 right way?

sorry, can't help you here.

- --
BOFH excuse #309:

firewall needs cooling
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFCNyB/C/PVm5+NVoYRAtRoAJ0WYPOBvNsz+LFe0r8xYfF8zEDpEACgpU/C
8NFhJ0YlSom+a9wHf7tStBk=
=hGXK
-----END PGP SIGNATURE-----