* Re: is_tree_inode: node level X does not match to the expected one Y
2003-09-15 11:43 is_tree_inode: node level X does not match to the expected one Y Martin Wilck
@ 2003-09-15 12:02 ` Oleg Drokin
0 siblings, 0 replies; 2+ messages in thread
From: Oleg Drokin @ 2003-09-15 12:02 UTC (permalink / raw)
To: Martin Wilck; +Cc: Reiser FS Mailing List
Hello!
On Mon, Sep 15, 2003 at 01:43:19PM +0200, Martin Wilck wrote:
> We are currently trying to debug an error that occurs only with
> ReiserFS. The error occurs during a high-IO-load stress test
> (copy/compare test).
> Sep 8 19:20:55 rx203 kernel: vs-13050: reiserfs_update_sd: \
> i/o failure occurred trying to update [12 56 0x0 SD] stat data\
> is_tree_node: node level 2 does not match to the expected one 3
> Sep 8 19:20:55 rx203 kernel: vs-5150: search_by_key: \
> invalid format found in block 13924. Fsck?
I *think* I remember Chris described something looking exactly like what you are
describing some time ago (related to SuSE-speciufic datalogging patches), which is in fact
a rare race, as I remember.
I also remember there were some proposed patches, but probably they
were not included into that release?
I CC Chris because he certainly should know more info on the topic.
Chris: corresponding discussion happened on reiserfs-dev on April 3, 2002 with subject "check_internal_node Q"
And even more related discussion happened on March 03, 2003 with subject
"is_tree_node: node level 1 does not match to the expected one 2"
> Interestingly, when I stop test, unmount the volumes and do a
> reiserfsck, no errors are reported, the file systems seem to be intact.
And this seems to confirm my theory, too.
> - is it possible that this is _not_ a hardware error, but a reiser
> problem?
Yes, it is possible.
> - can anybody give me a simple explanation what is actually going wrong
> in the situation at hand?
As I remember, there was a race where we first remember the node level, then
some fs change occurs possibly changing node level, then we check nopde level
and it is wrong.
I attach a patch that was proposed by Chris at the time and you might want to try it,
or Chris may have better patch already.
Bye,
Oleg
> I haven't been able to trigger this on ia32, but I have triggered it on
> my x86_64 box just by running 4 copies of fsx-linux at once. I thought
> the key_in_buffer checks were supposed to find this kind of thing
> though.
This patch adds a few additional checks to make sure each buffer in the
path is in the tree and the path is rooted correctly. It also changes
search_by_key to drop the path and research when the tree height
changes.
Perhaps someone can find a better check for key_in_buffer that provides
these checks as well. I actually wanted to just drop the key_in_buffer
checks completely, and have search_by_key research any time the tree
changed, but that lead to vs-3050 deadlocks under load.
-chris
--- linux.ul.1/fs/reiserfs/stree.c 2003-03-06 09:16:36.000000000 -0500
+++ linux.ul/fs/reiserfs/stree.c 2003-03-06 09:15:44.000000000 -0500
@@ -338,7 +338,27 @@
return &MAX_KEY;
}
+/* check each buffer in the path to make sure it is still in the tree */
+int path_in_tree (const struct path *path,
+ const struct super_block * s)
+{
+ int n_path_offset = path->path_length;
+ struct buffer_head *bh;
+
+ if (PATH_OFFSET_PBUFFER(path, FIRST_PATH_ELEMENT_OFFSET)->b_blocknr !=
+ SB_ROOT_BLOCK(s))
+ {
+ return 0;
+ }
+ while (n_path_offset-- > FIRST_PATH_ELEMENT_OFFSET) {
+ bh = PATH_OFFSET_PBUFFER(path, n_path_offset);
+ if (!B_IS_IN_TREE(bh)) {
+ return 0;
+ }
+ }
+ return 1;
+}
/* Get delimiting key of the buffer at the path and its right neighbor. */
inline const struct key * get_rkey (
const struct path * p_s_chk_path,
@@ -403,7 +423,7 @@
if ( COMP_KEYS(get_rkey(p_s_chk_path, p_s_sb), p_s_key) != 1 )
/* p_s_key must be less than right delimitiing key */
return 0;
- return 1;
+ return path_in_tree(p_s_chk_path, p_s_sb);
}
@@ -664,6 +684,7 @@
int n_node_level, n_retval;
int right_neighbor_of_leaf_node;
int fs_gen;
+ int tree_height;
struct buffer_head *reada_bh[SEARCH_BY_KEY_READA];
unsigned long reada_blocks[SEARCH_BY_KEY_READA];
int reada_count = 0;
@@ -699,6 +720,7 @@
/* prep path to have another element added to it. */
p_s_last_element = PATH_OFFSET_PELEMENT(p_s_search_path, ++p_s_search_path->path_length);
fs_gen = get_generation (p_s_sb);
+ tree_height = SB_TREE_HEIGHT(p_s_sb);
expected_level --;
/* schedule read of right neighbors */
@@ -724,7 +746,8 @@
to search is still in the tree rooted from the current buffer. If
not then repeat search from the root. */
if ( fs_changed (fs_gen, p_s_sb) &&
- (!B_IS_IN_TREE (p_s_bh) || !key_in_buffer(p_s_search_path, p_s_key, p_s_sb)) ) {
+ (tree_height != SB_TREE_HEIGHT(p_s_sb) ||
+ !B_IS_IN_TREE (p_s_bh) || !key_in_buffer(p_s_search_path, p_s_key, p_s_sb)) ) {
PROC_INFO_INC( p_s_sb, search_by_key_restarted );
PROC_INFO_INC( p_s_sb, sbk_restarted[ expected_level - 1 ] );
decrement_counters_in_path(p_s_search_path);
^ permalink raw reply [flat|nested] 2+ messages in thread