Reiser4 crash 2.6.16-mm1

Reiser4 crash 2.6.16-mm1

[Joe Feise]

Hi, 

I had an interesting crash on my 2.6.16-mm1 machine earlier today.
I usually mount /usr/local readonly:
/dev/sda6 on /usr/local type reiser4 (ro)
However, since I wanted to update a sw package, I remounted it r/w.
The installation of the sw package failed with reiser4 errors. Sorry, I 
don't have a dmesg output at this point, I'll send that tonight when I'm 
back at the machine.
But anyway, it reported index errors, and suggested running fsck.
I then ran an fsck.reiser4 --build-fs, which finished without reporting 
errors.
In further tests, I mounted /usr/local rw from the start, and saw the 
installation fail again. After that, /usr/local/lib was no longer 
accessible.
To me, this looks like corruption of some in-memory data structures.
Sorry to not be of more help wrt dmesg output. But I was hoping this is 
already known somewhere... 

 -Joe

Re: Reiser4 crash 2.6.16-mm1

[Toby Thain]

On 27-Mar-06, at 4:41 PM, Joe Feise wrote:

> Hi,
> I had an interesting crash on my 2.6.16-mm1 machine earlier today.
> I usually mount /usr/local readonly:
> /dev/sda6 on /usr/local type reiser4 (ro)
> However, since I wanted to update a sw package, I remounted it r/w.
> The installation of the sw package failed with reiser4 errors.  
> Sorry, I don't have a dmesg output at this point, I'll send that  
> tonight when I'm back at the machine.
> But anyway, it reported index errors, and suggested running fsck.
> I then ran an fsck.reiser4 --build-fs, which finished without  
> reporting errors.
> In further tests, I mounted /usr/local rw from the start, and saw  
> the installation fail again. After that, /usr/local/lib was no  
> longer accessible.
> To me, this looks like corruption of some in-memory data structures.

Have you run a memtest lately?

--Toby


> Sorry to not be of more help wrt dmesg output. But I was hoping  
> this is already known somewhere...
> -Joe

Re: Reiser4 crash 2.6.16-mm1

[Joe Feise]

Toby Thain writes:
> 
> On 27-Mar-06, at 4:41 PM, Joe Feise wrote: 
> 
>> Hi,
>> I had an interesting crash on my 2.6.16-mm1 machine earlier today.
>> I usually mount /usr/local readonly:
>> /dev/sda6 on /usr/local type reiser4 (ro)
>> However, since I wanted to update a sw package, I remounted it r/w.
>> The installation of the sw package failed with reiser4 errors.  Sorry, I 
>> don't have a dmesg output at this point, I'll send that  tonight when I'm 
>> back at the machine.
>> But anyway, it reported index errors, and suggested running fsck.
>> I then ran an fsck.reiser4 --build-fs, which finished without  reporting 
>> errors.
>> In further tests, I mounted /usr/local rw from the start, and saw  the 
>> installation fail again. After that, /usr/local/lib was no  longer 
>> accessible.
>> To me, this looks like corruption of some in-memory data structures.
> 
> Have you run a memtest lately?

Thanks for the suggestion. I haven't run a memtest, but I don't really think 
that the memory is bad. The machine most likely would have had other issues 
if that was the case.
In any case, I will run a memtest tonight, just to make sure. 

 -Joe

Re: Reiser4 crash 2.6.16-mm1

[Joe Feise]

[-- Attachment #1: Type: text/plain, Size: 1075 bytes --]

Joe Feise wrote on 03/27/06 13:41:

> Hi, 
> 
> I had an interesting crash on my 2.6.16-mm1 machine earlier today.
> I usually mount /usr/local readonly:
> /dev/sda6 on /usr/local type reiser4 (ro)
> However, since I wanted to update a sw package, I remounted it r/w.
> The installation of the sw package failed with reiser4 errors. Sorry, I 
> don't have a dmesg output at this point, I'll send that tonight when I'm 
> back at the machine.
> But anyway, it reported index errors, and suggested running fsck.
> I then ran an fsck.reiser4 --build-fs, which finished without reporting 
> errors.
> In further tests, I mounted /usr/local rw from the start, and saw the 
> installation fail again. After that, /usr/local/lib was no longer 
> accessible.
> To me, this looks like corruption of some in-memory data structures.
> Sorry to not be of more help wrt dmesg output. But I was hoping this is 
> already known somewhere... 



The dmesg output is attached.
If it matters, the command is make install in SpamAssassin 3.1.1.
Oh, and memtest didn't report any errors.

-Joe


[-- Attachment #2: dmesg.out --]
[-- Type: text/plain, Size: 15637 bytes --]

4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cut_tree_object (fs/reiser4/tree.c:1793)[nikita-2861]:
WARNING: failure: -5
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cut_tree_object (fs/reiser4/tree.c:1793)[nikita-2861]:
WARNING: failure: -5
<4>reiser4[perl5.8.6(10861)]: delete_object_unix_file (fs/reiser4/plugin/file/file.c:2987)[]:
WARNING: failed to truncate file (125991) on removal: -5
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cut_tree_object (fs/reiser4/tree.c:1793)[nikita-2861]:
WARNING: failure: -5
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cut_tree_object (fs/reiser4/tree.c:1793)[nikita-2861]:
WARNING: failure: -5
<4>reiser4[perl5.8.6(10861)]: delete_object_unix_file (fs/reiser4/plugin/file/file.c:2987)[]:
WARNING: failed to truncate file (125992) on removal: -5
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cut_tree_object (fs/reiser4/tree.c:1793)[nikita-2861]:
WARNING: failure: -5
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cut_tree_object (fs/reiser4/tree.c:1793)[nikita-2861]:
WARNING: failure: -5
<4>reiser4[perl5.8.6(10861)]: delete_object_unix_file (fs/reiser4/plugin/file/file.c:2987)[]:
WARNING: failed to truncate file (125993) on removal: -5
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cut_tree_object (fs/reiser4/tree.c:1793)[nikita-2861]:
WARNING: failure: -5
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cut_tree_object (fs/reiser4/tree.c:1793)[nikita-2861]:
WARNING: failure: -5
<4>reiser4[perl5.8.6(10861)]: delete_object_unix_file (fs/reiser4/plugin/file/file.c:2987)[]:
WARNING: failed to truncate file (125994) on removal: -5
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cut_tree_object (fs/reiser4/tree.c:1793)[nikita-2861]:
WARNING: failure: -5
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cut_tree_object (fs/reiser4/tree.c:1793)[nikita-2861]:
WARNING: failure: -5
<4>reiser4[perl5.8.6(10861)]: delete_object_unix_file (fs/reiser4/plugin/file/file.c:2987)[]:
WARNING: failed to truncate file (125995) on removal: -5
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cut_tree_object (fs/reiser4/tree.c:1793)[nikita-2861]:
WARNING: failure: -5
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cut_tree_object (fs/reiser4/tree.c:1793)[nikita-2861]:
WARNING: failure: -5
<4>reiser4[perl5.8.6(10861)]: delete_object_unix_file (fs/reiser4/plugin/file/file.c:2987)[]:
WARNING: failed to truncate file (125996) on removal: -5
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cut_tree_object (fs/reiser4/tree.c:1793)[nikita-2861]:
WARNING: failure: -5
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cut_tree_object (fs/reiser4/tree.c:1793)[nikita-2861]:
WARNING: failure: -5
<4>reiser4[perl5.8.6(10861)]: delete_object_unix_file (fs/reiser4/plugin/file/file.c:2987)[]:
WARNING: failed to truncate file (125997) on removal: -5
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cut_tree_object (fs/reiser4/tree.c:1793)[nikita-2861]:
WARNING: failure: -5
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cut_tree_object (fs/reiser4/tree.c:1793)[nikita-2861]:
WARNING: failure: -5
<4>reiser4[perl5.8.6(10861)]: delete_object_unix_file (fs/reiser4/plugin/file/file.c:2987)[]:
WARNING: failed to truncate file (125998) on removal: -5
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cut_tree_object (fs/reiser4/tree.c:1793)[nikita-2861]:
WARNING: failure: -5
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cut_tree_object (fs/reiser4/tree.c:1793)[nikita-2861]:
WARNING: failure: -5
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: key_warning (fs/reiser4/plugin/file_plugin_common.c:514)[nikita-717]:
WARNING: Error for inode 77957 (-5)
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: key_warning (fs/reiser4/plugin/file_plugin_common.c:514)[nikita-717]:
WARNING: Error for inode 77957 (-5)
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: key_warning (fs/reiser4/plugin/file_plugin_common.c:514)[nikita-717]:
WARNING: Error for inode 77957 (-5)
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: key_warning (fs/reiser4/plugin/file_plugin_common.c:514)[nikita-717]:
WARNING: Error for inode 77957 (-5)
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: key_warning (fs/reiser4/plugin/file_plugin_common.c:514)[nikita-717]:
WARNING: Error for inode 77957 (-5)
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: key_warning (fs/reiser4/plugin/file_plugin_common.c:514)[nikita-717]:
WARNING: Error for inode 77957 (-5)
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: key_warning (fs/reiser4/plugin/file_plugin_common.c:514)[nikita-717]:
WARNING: Error for inode 77957 (-5)
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: key_warning (fs/reiser4/plugin/file_plugin_common.c:514)[nikita-717]:
WARNING: Error for inode 78025 (-5)
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: key_warning (fs/reiser4/plugin/file_plugin_common.c:514)[nikita-717]:
WARNING: Error for inode 78025 (-5)
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: key_warning (fs/reiser4/plugin/file_plugin_common.c:514)[nikita-717]:
WARNING: Error for inode 78025 (-5)
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: key_warning (fs/reiser4/plugin/file_plugin_common.c:514)[nikita-717]:
WARNING: Error for inode 78025 (-5)
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: key_warning (fs/reiser4/plugin/file_plugin_common.c:514)[nikita-717]:
WARNING: Error for inode 78025 (-5)
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: key_warning (fs/reiser4/plugin/file_plugin_common.c:514)[nikita-717]:
WARNING: Error for inode 78025 (-5)
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: key_warning (fs/reiser4/plugin/file_plugin_common.c:514)[nikita-717]:
WARNING: Error for inode 78025 (-5)
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cut_tree_object (fs/reiser4/tree.c:1793)[nikita-2861]:
WARNING: failure: -5
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: key_warning (fs/reiser4/plugin/file_plugin_common.c:514)[nikita-717]:
WARNING: Error for inode 1097533 (-5)
<4>reiser4[perl5.8.6(10861)]: write_sd_by_inode_common (fs/reiser4/plugin/file_plugin_common.c:34)[nikita-2221]:
WARNING: Failed to save sd for 1097533: -5
<4>reiser4[perl5.8.6(10861)]: do_create_vfs_child (fs/reiser4/plugin/inode_ops.c:685)[nikita-2219]:
WARNING: Failed to create sd for 1097533
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: key_warning (fs/reiser4/plugin/file_plugin_common.c:514)[nikita-717]:
WARNING: Error for inode 78027 (-5)
<4>reiser4[perl5.8.6(10861)]: write_sd_by_inode_common (fs/reiser4/plugin/file_plugin_common.c:34)[nikita-2221]:
WARNING: Failed to save sd for 78027: -5
<4>reiser4[perl5.8.6(10861)]: reiser4_dirty_inode (fs/reiser4/super_ops.c:177)[]:
WARNING: failed to dirty inode for 78027: -5
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: key_warning (fs/reiser4/plugin/file_plugin_common.c:514)[nikita-717]:
WARNING: Error for inode 78027 (-5)
<4>reiser4[perl5.8.6(10861)]: write_sd_by_inode_common (fs/reiser4/plugin/file_plugin_common.c:34)[nikita-2221]:
WARNING: Failed to save sd for 78027: -5
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: key_warning (fs/reiser4/plugin/file_plugin_common.c:514)[nikita-717]:
WARNING: Error for inode 78027 (-5)
<4>reiser4[perl5.8.6(10861)]: write_sd_by_inode_common (fs/reiser4/plugin/file_plugin_common.c:34)[nikita-2221]:
WARNING: Failed to save sd for 78027: -5
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: key_warning (fs/reiser4/plugin/file_plugin_common.c:514)[nikita-717]:
WARNING: Error for inode 1097534 (-5)
<4>reiser4[perl5.8.6(10861)]: write_sd_by_inode_common (fs/reiser4/plugin/file_plugin_common.c:34)[nikita-2221]:
WARNING: Failed to save sd for 1097534: -5
<4>reiser4[perl5.8.6(10861)]: do_create_vfs_child (fs/reiser4/plugin/inode_ops.c:685)[nikita-2219]:
WARNING: Failed to create sd for 1097534
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: setattr_truncate (fs/reiser4/plugin/file/file.c:2888)[vs-1588]:
WARNING: truncate_file failed: oid 99559, old size 8640, new size 0, retval -5
<4>reiser4[perl5.8.6(10861)]: cbk_level_lookup (fs/reiser4/search.c:971)[vs-3533]:
WARNING: Keys are inconsistent. Fsck?
<4>reiser4[perl5.8.6(10861)]: cut_tree_object (fs/reiser4/tree.c:1793)[nikita-2861]:
WARNING: failure: -5
<4>reiser4[perl5.8.6(10861)]: setattr_truncate (fs/reiser4/plugin/file/file.c:2896)[nikita-3417]:
WARNING: Cannot kill safelink 99559: -5

Re: Reiser4 crash 2.6.16-mm1

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 544 bytes --]

On Mon, 27 Mar 2006 14:32:14 PST, Joe Feise said:

> Thanks for the suggestion. I haven't run a memtest, but I don't really think 
> that the memory is bad. The machine most likely would have had other issues 
> if that was the case.

You'd be *amazed*.  Intermittently weak memory (especially if it's just one bad
bit) can manifest in the most odd ways.

In fact, if you think about it, if it's bad memory, your trashed reiser4
partition could very well *be* that "would have had other issues" that you
said you'd see if it was bad memory. ;)

[-- Attachment #2: Type: application/pgp-signature, Size: 228 bytes --]

Re: Reiser4 crash 2.6.16-mm1

[Toby Thain]

On 27-Mar-06, at 11:39 PM, Valdis.Kletnieks@vt.edu wrote:

> On Mon, 27 Mar 2006 14:32:14 PST, Joe Feise said:
>
>> Thanks for the suggestion. I haven't run a memtest, but I don't  
>> really think
>> that the memory is bad. The machine most likely would have had  
>> other issues
>> if that was the case.
>
> You'd be *amazed*.  Intermittently weak memory (especially if it's  
> just one bad
> bit) can manifest in the most odd ways.


I tend to agree. I spent weeks/months chasing down what I thought was  
a chipset bug, when it was bad RAM. Disk reads (and probably writes)  
were being corrupted and the kernel did not know about it. Was very  
frustrating ... until I figured out the real problem.

Joe, did you soak the test at least overnight? Have you done any  
heavy compiles (like building X11, or gcc) lately? Compilers are  
often the canaries in the mine, when it comes to RAM. I'm not saying  
this is your problem but it would be good to rule out first.

--Toby

>
> In fact, if you think about it, if it's bad memory, your trashed  
> reiser4
> partition could very well *be* that "would have had other issues"  
> that you
> said you'd see if it was bad memory. ;)

Re: Reiser4 crash 2.6.16-mm1

[Joachim Feise]

Toby Thain wrote on 03/27/06 22:34:

> On 27-Mar-06, at 11:39 PM, Valdis.Kletnieks@vt.edu wrote:
> 
>> On Mon, 27 Mar 2006 14:32:14 PST, Joe Feise said:
>>
>>> Thanks for the suggestion. I haven't run a memtest, but I don't  
>>> really think
>>> that the memory is bad. The machine most likely would have had  
>>> other issues
>>> if that was the case.
>> You'd be *amazed*.  Intermittently weak memory (especially if it's  
>> just one bad
>> bit) can manifest in the most odd ways.
> 
> 
> I tend to agree. I spent weeks/months chasing down what I thought was  
> a chipset bug, when it was bad RAM. Disk reads (and probably writes)  
> were being corrupted and the kernel did not know about it. Was very  
> frustrating ... until I figured out the real problem.
> 
> Joe, did you soak the test at least overnight? Have you done any  
> heavy compiles (like building X11, or gcc) lately? Compilers are  
> often the canaries in the mine, when it comes to RAM. I'm not saying  
> this is your problem but it would be good to rule out first.


This is a production machine that I can't take offline for too long.
But yes, I have compiled the kernel on another reiser4 partition over night,
without problems.
If this was a memory problem, it would indeed manifest itself in other areas
with more or less random errors. The fact that it does not indicates to me that
this is a fs problem. So, at this point I am ruling out a memory issue.

-Joe

Re: Reiser4 crash 2.6.16-mm1

[Toby Thain]

On 28-Mar-06, at 10:34 AM, Joachim Feise wrote:

> Toby Thain wrote on 03/27/06 22:34:
>
>> On 27-Mar-06, at 11:39 PM, Valdis.Kletnieks@vt.edu wrote:
>>
>>> On Mon, 27 Mar 2006 14:32:14 PST, Joe Feise said:
>>>
>>>> Thanks for the suggestion. I haven't run a memtest, but I don't
>>>> really think
>>>> that the memory is bad. The machine most likely would have had
>>>> other issues
>>>> if that was the case.
>>> You'd be *amazed*.  Intermittently weak memory (especially if it's
>>> just one bad
>>> bit) can manifest in the most odd ways.
>>
>>
>> I tend to agree. I spent weeks/months chasing down what I thought was
>> a chipset bug, when it was bad RAM. Disk reads (and probably writes)
>> were being corrupted and the kernel did not know about it. Was very
>> frustrating ... until I figured out the real problem.
>>
>> Joe, did you soak the test at least overnight? Have you done any
>> heavy compiles (like building X11, or gcc) lately? Compilers are
>> often the canaries in the mine, when it comes to RAM. I'm not saying
>> this is your problem but it would be good to rule out first.
>
>
> This is a production machine that I can't take offline for too long.
> But yes, I have compiled the kernel on another reiser4 partition  
> over night,
> without problems.
> If this was a memory problem, it would indeed manifest itself in  
> other areas
> with more or less random errors. The fact that it does not  
> indicates to me that
> this is a fs problem. So, at this point I am ruling out a memory  
> issue.


I agree with all of the above, except the "random errors" part.  
Depending on the type of fault, it may manifest only under very  
specific but repeatable conditions, and never be seen in ordinary  
workload. Other faults, of course, will manifest in many contexts and  
apparently randomly. But I guess this is OT by now. :-)

--Toby

>
> -Joe

Re: Reiser4 crash 2.6.16-mm1

[Jonathan Briggs]

[-- Attachment #1: Type: text/plain, Size: 682 bytes --]

On Tue, 2006-03-28 at 07:34 -0800, Joachim Feise wrote:
[...]
> This is a production machine that I can't take offline for too long.
> But yes, I have compiled the kernel on another reiser4 partition over night,
> without problems.
> If this was a memory problem, it would indeed manifest itself in other areas
> with more or less random errors. The fact that it does not indicates to me that
> this is a fs problem. So, at this point I am ruling out a memory issue.

And if it's a production machine, it is using ECC RAM, I would hope.  If
it is, memory problems (unreported ones, anyway) are very, very
unlikely.
-- 
Jonathan Briggs <jbriggs@esoft.com>
eSoft, Inc.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 191 bytes --]

Re: Reiser4 crash 2.6.16-mm1

[Hans Reiser]

Jonathan Briggs wrote:

>On Tue, 2006-03-28 at 07:34 -0800, Joachim Feise wrote:
>[...]
>  
>
>>This is a production machine that I can't take offline for too long.
>>But yes, I have compiled the kernel on another reiser4 partition over night,
>>without problems.
>>If this was a memory problem, it would indeed manifest itself in other areas
>>with more or less random errors. The fact that it does not indicates to me that
>>this is a fs problem. So, at this point I am ruling out a memory issue.
>>    
>>
>
>And if it's a production machine, it is using ECC RAM, I would hope.  If
>it is, memory problems (unreported ones, anyway) are very, very
>unlikely.
>  
>
Jonathan, be merciful, ECC ram last I checked is twice the cost of
regular and the mobs cost more too.  (I am sure the cost to produce is <
15% more, which makes it a great pity Intel does not standardize on
requiring it and force it to be cheap)  Some folks need to save money. 
Yeah, I know, this time it may have cost him more in cost of his time
but we are all just assuming it is memory.  Unfortunately, unless he
checks it or we see an identical error message from another user with
checked memory, or vs tells me he sees a flaw in the code, we need to
assume it is memory.

Reiser4 is more stable than the hardware it runs on.  At some point it
will be more stable than the hardware users are sure isn't the problem,
like V3 became.;-)

Hans

Re: Reiser4 crash 2.6.16-mm1

[Jonathan Briggs]

[-- Attachment #1: Type: text/plain, Size: 1356 bytes --]

On Tue, 2006-03-28 at 12:08 -0800, Hans Reiser wrote:
> Jonathan Briggs wrote:
[...]
> >And if it's a production machine, it is using ECC RAM, I would hope.  If
> >it is, memory problems (unreported ones, anyway) are very, very
> >unlikely.
> >  
> >
> Jonathan, be merciful, ECC ram last I checked is twice the cost of
> regular and the mobs cost more too.  (I am sure the cost to produce is <
> 15% more, which makes it a great pity Intel does not standardize on
> requiring it and force it to be cheap)  Some folks need to save money. 
> Yeah, I know, this time it may have cost him more in cost of his time
> but we are all just assuming it is memory.  Unfortunately, unless he
> checks it or we see an identical error message from another user with
> checked memory, or vs tells me he sees a flaw in the code, we need to
> assume it is memory.
[...]

Yes, I know. :)

But for a production machine that is "producing" something of value, the
extra cost should not be an issue.  RAM errors are so subtle and so hard
to find that ECC is of far more value than RAID.  It is obvious when
your disk fails.

An extra high bit in a credit transaction could cost you $16,384 and you
might not ever realize what happened. :)

Anyway, off topic, but ECC is highly recommended.
-- 
Jonathan Briggs <jbriggs@esoft.com>
eSoft, Inc.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 191 bytes --]

Re: Reiser4 crash 2.6.16-mm1

[Joe Feise]

Jonathan Briggs writes: 

> On Tue, 2006-03-28 at 07:34 -0800, Joachim Feise wrote:
> [...]
>> This is a production machine that I can't take offline for too long.
>> But yes, I have compiled the kernel on another reiser4 partition over night,
>> without problems.
>> If this was a memory problem, it would indeed manifest itself in other areas
>> with more or less random errors. The fact that it does not indicates to me that
>> this is a fs problem. So, at this point I am ruling out a memory issue.
> 
> And if it's a production machine, it is using ECC RAM, I would hope.  If

Indeed, it does. Dell gets that right on server machines, at least... 

I have now recreated the fs on that partition, and played the data back from 
backup, and it works fine. I have made a copy of the old partition with dd, 
to be able to run some more tests when I have time. 

 -Joe

Re: Reiser4 crash 2.6.16-mm1

[Joe Feise]

Hans Reiser writes: 

> Jonathan Briggs wrote: 
> 
>>On Tue, 2006-03-28 at 07:34 -0800, Joachim Feise wrote:
>>[...]
>>   
>>
>>>This is a production machine that I can't take offline for too long.
>>>But yes, I have compiled the kernel on another reiser4 partition over night,
>>>without problems.
>>>If this was a memory problem, it would indeed manifest itself in other areas
>>>with more or less random errors. The fact that it does not indicates to me that
>>>this is a fs problem. So, at this point I am ruling out a memory issue.
>>>     
>>>
>>
>>And if it's a production machine, it is using ECC RAM, I would hope.  If
>>it is, memory problems (unreported ones, anyway) are very, very
>>unlikely.
>>   
>>
> Jonathan, be merciful, ECC ram last I checked is twice the cost of
> regular and the mobs cost more too.  (I am sure the cost to produce is <
> 15% more, which makes it a great pity Intel does not standardize on
> requiring it and force it to be cheap)  Some folks need to save money. 
> Yeah, I know, this time it may have cost him more in cost of his time
> but we are all just assuming it is memory.  Unfortunately, unless he
> checks it or we see an identical error message from another user with
> checked memory, or vs tells me he sees a flaw in the code, we need to
> assume it is memory.

The machine is using ECC memory. Geez, I know what I need for a server...
 From the Dell invoice:
512MB DDR2, 400MHz,2X256MB ECC 1R DIMMs for PowerEdge SC420
Recreating the partition solved the problem. So to me it sure looks like fs 
corruption.
I have sent the dmesg output earlier. 

 -Joe

Re: Reiser4 crash 2.6.16-mm1

[Hans Reiser]

Joe Feise wrote:

> Jonathan Briggs writes:
>
>> On Tue, 2006-03-28 at 07:34 -0800, Joachim Feise wrote:
>> [...]
>>
>>> This is a production machine that I can't take offline for too long.
>>> But yes, I have compiled the kernel on another reiser4 partition
>>> over night,
>>> without problems.
>>> If this was a memory problem, it would indeed manifest itself in
>>> other areas
>>> with more or less random errors. The fact that it does not indicates
>>> to me that
>>> this is a fs problem. So, at this point I am ruling out a memory issue.
>>
>>
>> And if it's a production machine, it is using ECC RAM, I would hope.  If
>
>
> Indeed, it does. Dell gets that right on server machines, at least...
> I have now recreated the fs on that partition, and played the data
> back from backup, and it works fine. I have made a copy of the old
> partition with dd, to be able to run some more tests when I have time.
> -Joe
>
>
Well, if it is ECC RAM, then we should probably assume it is a bug. 
Sigh.  Vs, can you look at it?

Re: Reiser4 crash 2.6.16-mm1

[Gregory Maxwell]

On 3/28/06, Jonathan Briggs <jbriggs@esoft.com> wrote:

> But for a production machine that is "producing" something of value, the
> extra cost should not be an issue.  RAM errors are so subtle and so hard
> to find that ECC is of far more value than RAID.  It is obvious when
> your disk fails.
>
> An extra high bit in a credit transaction could cost you $16,384 and you
> might not ever realize what happened. :)
>
> Anyway, off topic, but ECC is highly recommended.

And with the amount of memory that people are putting in modern system
1 bit events should be happening on a approx weekly basis.

ECC may be more expensive but it doesn't make memory more expensive
than it was just a few years ago.... you really should have it.

But this has gone far offtopic.

Re: Reiser4 crash 2.6.16-mm1

[Sergey Ivanov]

Joe Feise wrote:
> 
> The machine is using ECC memory. Geez, I know what I need for a server...
> From the Dell invoice:
> 512MB DDR2, 400MHz,2X256MB ECC 1R DIMMs for PowerEdge SC420
> Recreating the partition solved the problem. So to me it sure looks like
> fs corruption.
> I have sent the dmesg output earlier.
Joe, I saw machines with ECC hardware (both motherboards and memory
chips) with ECC disabled in BIOS. Check please, if it's enabled for your
machine.
-- 
	Sergey Ivanov.

Re: Reiser4 crash 2.6.16-mm1

[Joe Feise]

Gregory Maxwell wrote on 03/28/06 13:22:

> On 3/28/06, Jonathan Briggs <jbriggs@esoft.com> wrote:
> 
>> But for a production machine that is "producing" something of value, the
>> extra cost should not be an issue.  RAM errors are so subtle and so hard
>> to find that ECC is of far more value than RAID.  It is obvious when
>> your disk fails.
>>
>> An extra high bit in a credit transaction could cost you $16,384 and you
>> might not ever realize what happened. :)
>>
>> Anyway, off topic, but ECC is highly recommended.
> 
> And with the amount of memory that people are putting in modern system
> 1 bit events should be happening on a approx weekly basis.
> 
> ECC may be more expensive but it doesn't make memory more expensive
> than it was just a few years ago.... you really should have it.


I agree, and the system in question has ECC RAM.
The additional cost is not such a big issue anyway... I have SCSI disks (no
RAID, though) in there, and I could get at least 3 times the capacity for the
same price if I was using IDE...

-Joe

Re: Reiser4 crash 2.6.16-mm1

[Joe Feise]

Sergey Ivanov wrote on 03/28/06 13:52:

> Joe Feise wrote:
>> The machine is using ECC memory. Geez, I know what I need for a server...
>> From the Dell invoice:
>> 512MB DDR2, 400MHz,2X256MB ECC 1R DIMMs for PowerEdge SC420
>> Recreating the partition solved the problem. So to me it sure looks like
>> fs corruption.
>> I have sent the dmesg output earlier.
> Joe, I saw machines with ECC hardware (both motherboards and memory
> chips) with ECC disabled in BIOS. Check please, if it's enabled for your
> machine.


It is enabled. The Dell BIOS actually doesn't even allow to change that.

-Joe