* reiserfs3 bug? @ 2006-04-05 14:39 Jason Keltz 2006-04-20 13:27 ` reiserfs3 bug? 2.4.32 Jason Keltz 0 siblings, 1 reply; 4+ messages in thread From: Jason Keltz @ 2006-04-05 14:39 UTC (permalink / raw) To: reiserfs-list Our Linux 2.4.32 NFS fileserver exports 4 reiserfs 3.6 filesystems to a whole bunch of hosts. Somewhere between every 6-30 days, NFS suddenly seems to "hang" (ie. all the hosts gets the "nfs server not responding" message). The server is still up (we can ssh to it/etc). Up to this point, we've thought it was a bug in nfs. We recently installed SGI kdb (kernel debugger) to help with debugging the problem, and we're wondering now whether it is actually reiserfs related. We need to get the output from several crashes in order to do more debugging, although we believe the problem is probably the same each time. In a normal state, the WCHAN column on "ps" output lists the nfsd and [kreiserfsd] processes as "end". When the system gets into this state, both the nfsd and [kreiserfsd] processes report "down" for WCHAN. (I can imagine that if kreiserfsd and nfs are both hanging on the same lock, bad things could happen..) The backtrace when the problem occurs for kreiserfsd yields: Mar 28 13:04:24 0xf635a000 247 1 0 1 D 0xf635a370 kreiserfsd Mar 28 13:04:24 ESP EIP Function (args) Mar 28 13:04:24 0xf635bef8 0xc011b144 schedule+0x2b4 (0xc0452e40, 0x0, 0xf635a000, 0xd4a10970, 0xd4a10970) Mar 28 13:04:24 kernel .text 0xc0100000 0xc011ae90 0xc011b3d0 Mar 28 13:04:24 0xf635bf40 0xc014680e __wait_on_buffer+0x6e (0xd4a10920, 0x9e0, 0xf635bf90, 0xc2937000, 0xf8ac6000) Mar 28 13:04:24 kernel .text 0xc0100000 0xc01467a0 0xc0146840 Mar 28 13:04:24 0xf635bf68 0xf8943e79 [reiserfs]flush_commit_list+0x3e9 Mar 28 13:04:24 reiserfs .text 0xf8921060 0xf8943a90 0xf8943f60 Mar 28 13:04:24 0xf635bfa8 0xf894801d [reiserfs]flush_async_commits+0x3d (0xf7576800, 0xdd667cc0, 0xf635bfd8, 0xf635bfdc, 0x20) Mar 28 13:04:24 reiserfs .text 0xf8921060 0xf8947fe0 0xf8948020 Mar 28 13:04:25 0xf635bfb8 0xf894652b [reiserfs]reiserfs_journal_commit_thread+0x1db Mar 28 13:04:25 reiserfs .text 0xf8921060 0xf8946350 0xf89465f0 Mar 28 13:04:25 0xf635bff4 0xc010741e arch_kernel_thread+0x2e Mar 28 13:04:25 kernel .text 0xc0100000 0xc01073f0 0xc0107430 single stepping on the processor after the problem reveals that the system is "idle"/not doing anything else with this. I won't bother including the output of the backtrace of the 256 nfs processes on our fileserver here, but they probably give a lot more of the story. If you are interested, please see this link for the full details: http://www.cs.yorku.ca/~jas/fileserver If anyone has any ideas, or anywhere we could insert debugging code iin order to help solve this problem, we would *really* appreciate your help! We recently upgraded from 2.4.26 to 2.4.32 in the hopes that the bug would have been fixed, but it didn't make any difference. ps: A few times, when we issue the "reboot" command, the systems get "unstuck" (systems get "nfs ok") just before the system reboots... whatever is stuck seems to get unstuck for a moment before the system is rebooted. Thanks.. Jason Keltz jas@cs.yorku.ca ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: reiserfs3 bug? 2.4.32 2006-04-05 14:39 reiserfs3 bug? Jason Keltz @ 2006-04-20 13:27 ` Jason Keltz 2006-04-21 7:47 ` Vladimir V. Saveliev 0 siblings, 1 reply; 4+ messages in thread From: Jason Keltz @ 2006-04-20 13:27 UTC (permalink / raw) To: Jason Keltz; +Cc: reiserfs-list On April 5, I had sent an email to the list about a problem that we were having on our systems that seemed to be reiserfs related. Unfortunately, I didn't get any response. I'm trying one more time with other information in the hope that someone might be able to assist us in solving this problem. We're having problems with our file server crashing every 6-30 days. I can't remember the last time it stayed up longer than 30 days. I've recently installed kdb in order to get more debugging information in the hopes that I might be able to get assistance in solving this problem. The server crashed for the second time yesterday since I got kdb installed (18 days since the last time). I've attached some minimal output from kdb (kreiserfsd process, rpc.mountd, lockd, and then 2 nfsd processes). Any assistance would be appreciated. I e-mailed Neil Brown of NFS server fame, and he confirmed that the problem was not an NFS server problem, but a reiserfs one. Neil gave me this excellent description of the problem: The deadlock is happening in flushing out the journal. Any process that tries to write will need to wait for that journal flush to complete, which it won't. One nfsd thread is blocked waiting to write. It holds a read lock on the nfsd exports table. There is an 'rpc.mountd' process that is trying to get a write lock on the exports table. It will block until the read lock is released. The other nfsd threads are trying to get a read lock. That won't succeed until mountd gets it's write lock and releases it. Thus all but one nfsd threads are blocked by mountd which is blocked by the remaining nfsd which is blocked by kreiserfsd. Special things about our nfs server: The server runs stock kernel 2.4.32 with kdb patch and reiserfs quota patch. NFS filesystems being exported are all reiserfs3. It is a dual processor system running the SMP kernel. All exported filesystems are on a 3ware raid card. The kdb logs: Apr 19 13:39:36 0xf67fc000 247 1 0 1 D 0xf67fc370 kreiserfsd Apr 19 13:39:36 ESP EIP Function (args) Apr 19 13:39:36 0xf67fde98 0xc011b144 schedule+0x2b4 (0xc0452e40, 0x0, 0xf67fc000, 0xc91f8a10, 0xc91f8a10) Apr 19 13:39:36 kernel .text 0xc0100000 0xc011ae90 0xc011b3d0 Apr 19 13:39:36 0xf67fdee0 0xc014680e __wait_on_buffer+0x6e (0xc91f89c0, 0x1601, 0xf67fdf30, 0x1, 0xeeee4000) Apr 19 13:39:36 kernel .text 0xc0100000 0xc01467a0 0xc0146840 Apr 19 13:39:36 0xf67fdf08 0xf8943e79 [reiserfs]flush_commit_list+0x3e9 (0xf69ab800, 0xeae57b20, 0x0) Apr 19 13:39:36 reiserfs .text 0xf8921060 0xf8943a90 0xf8943f60 Apr 19 13:39:36 0xf67fdf48 0xf8943a41 [reiserfs]flush_older_commits+0x91 (0xf69ab800, 0xeae570a0, 0xebc80578, 0xf8ac6000, 0xf 6a2373c) Apr 19 13:39:37 reiserfs .text 0xf8921060 0xf89439b0 0xf8943a90 Apr 19 13:39:37 0xf67fdf68 0xf8943b24 [reiserfs]flush_commit_list+0x94 Apr 19 13:39:37 reiserfs .text 0xf8921060 0xf8943a90 0xf8943f60 Apr 19 13:39:37 0xf67fdfa8 0xf894801d [reiserfs]flush_async_commits+0x3d (0xf69ab800, 0xd17ebee0, 0xf67fdfd8, 0xf67fdfdc, 0x2 0) Apr 19 13:39:37 reiserfs .text 0xf8921060 0xf8947fe0 0xf8948020 Apr 19 13:39:37 0xf67fdfb8 0xf894652b [reiserfs]reiserfs_journal_commit_thread+0x1db Apr 19 13:39:37 reiserfs .text 0xf8921060 0xf8946350 0xf89465f0 Apr 19 13:39:37 0xf67fdff4 0xc010741e arch_kernel_thread+0x2e Apr 19 13:39:37 kernel .text 0xc0100000 0xc01073f0 0xc0107430 Apr 19 13:39:47 0xeeffe000 993 1 0 0 S 0xeeffe370 rpc.mountd Apr 19 13:39:47 ESP EIP Function (args) Apr 19 13:39:47 0xeefffe04 0xc011b144 schedule+0x2b4 (0x0, 0xeeffe000, 0xf8d30164, 0xee98ff88, 0xeeffe000) Apr 19 13:39:47 kernel .text 0xc0100000 0xc011ae90 0xc011b3d0 Apr 19 13:39:48 0xeefffe4c 0xc011b78f interruptible_sleep_on+0x4f Apr 19 13:39:48 kernel .text 0xc0100000 0xc011b740 0xc011b7d0 Apr 19 13:39:48 0xeefffe6c 0xf8d24d8a [nfsd]exp_writelock+0x5a Apr 19 13:39:48 nfsd .text 0xf8d1d060 0xf8d24d30 0xf8d24e30 Apr 19 13:39:48 0xeefffe74 0xf8d244e6 [nfsd]exp_export+0x76 (0xdff64004, 0xbfffd000, 0x814, 0xc0151015, 0xcf62dc20) Apr 19 13:39:48 nfsd .text 0xf8d1d060 0xf8d24470 0xf8d24880 Apr 19 13:39:48 0xeefffed4 0xf8d1da4e [nfsd]handle_sys_nfsservctl+0x23e (0x3, 0xbfffd000, 0x0, 0xeeffe000) Apr 19 13:39:48 nfsd .text 0xf8d1d060 0xf8d1d810 0xf8d1dc90 Apr 19 13:39:48 0xeeffffac 0xc0160ed6 sys_nfsservctl+0x76 (0x3, 0xbfffd000, 0x0, 0x420dabf7, 0x2) Apr 19 13:39:48 kernel .text 0xc0100000 0xc0160e60 0xc0160f4b Apr 19 13:39:48 0xeeffffc4 0xc0108f1f system_call+0x33 Apr 19 13:39:48 kernel .text 0xc0100000 0xc0108eec 0xc0108f24 Apr 19 13:40:11 0xeef70000 1021 1 0 1 D 0xeef70370 lockd Apr 19 13:40:11 ESP EIP Function (args) Apr 19 13:40:12 0xeef71f60 0xc011b144 schedule+0x2b4 (0x0, 0xeef70000, 0xeee93f88, 0xf8d30164, 0xeef70000) Apr 19 13:40:12 kernel .text 0xc0100000 0xc011ae90 0xc011b3d0 Apr 19 13:40:12 0xeef71fa8 0xc011b8af sleep_on+0x4f Apr 19 13:40:12 kernel .text 0xc0100000 0xc011b860 0xc011b8f0 Apr 19 13:40:12 0xeef71fc8 0xf8d24d0a [nfsd]exp_readlock+0x2a (0xc28ed2e0, 0xeefa7000, 0x7fffffff, 0xeef70000, 0x4789) Apr 19 13:40:12 nfsd .text 0xf8d1d060 0xf8d24ce0 0xf8d24d30 Apr 19 13:40:12 0xeef71fcc 0xf8d10149 [lockd]lockd+0x1c9 Apr 19 13:40:12 lockd .text 0xf8d0e060 0xf8d0ff80 0xf8d10260 Apr 19 13:40:12 0xeef71ff4 0xc010741e arch_kernel_thread+0x2e Apr 19 13:40:12 kernel .text 0xc0100000 0xc01073f0 0xc0107430 almost every nfsd was like this: Apr 19 13:39:49 0xef00a000 998 1 0 1 D 0xef00a370 nfsd Apr 19 13:39:49 ESP EIP Function (args) Apr 19 13:39:49 0xef00bf38 0xc011b144 schedule+0x2b4 (0x0, 0xef00a000, 0xeed05f88, 0xee997f88, 0x337) Apr 19 13:39:49 kernel .text 0xc0100000 0xc011ae90 0xc011b3d0 Apr 19 13:39:49 0xef00bf80 0xc011b8af sleep_on+0x4f Apr 19 13:39:49 kernel .text 0xc0100000 0xc011b860 0xc011b8f0 Apr 19 13:39:49 0xef00bfa0 0xf8d24d0a [nfsd]exp_readlock+0x2a (0xc28ed260, 0xf002d800, 0x7530, 0xb4fae97, 0xef00a000) Apr 19 13:39:49 nfsd .text 0xf8d1d060 0xf8d24ce0 0xf8d24d30 Apr 19 13:39:49 0xef00bfa4 0xf8d1d3a8 [nfsd]nfsd+0x1a8 Apr 19 13:39:49 nfsd .text 0xf8d1d060 0xf8d1d200 0xf8d1d580 Apr 19 13:39:49 0xef00bff4 0xc010741e arch_kernel_thread+0x2e Apr 19 13:39:49 kernel .text 0xc0100000 0xc01073f0 0xc0107430 and one like this: Apr 19 13:40:33 0xeeee4000 1043 1 0 0 D 0xeeee4370 nfsd Apr 19 13:40:33 ESP EIP Function (args) Apr 19 13:40:33 0xeeee5920 0xc011b144 schedule+0x2b4 (0x1, 0xeeee4000, 0xeae57b44, 0xeae57b44, 0x14ca5af6) Apr 19 13:40:33 kernel .text 0xc0100000 0xc011ae90 0xc011b3d0 Apr 19 13:40:33 0xeeee5968 0xc01079b3 __down+0x83 Apr 19 13:40:33 kernel .text 0xc0100000 0xc0107930 0xc0107a10 Apr 19 13:40:33 0xeeee598c 0xc0107b5c __down_failed+0x8 (0xf69ab800, 0xeae57b20, 0xeeee59c4, 0x0, 0xf7fda438) Apr 19 13:40:33 kernel .text 0xc0100000 0xc0107b54 0xc0107b60 Apr 19 13:40:34 0xeeee599c 0xf894969c [reiserfs].text.lock.journal+0x5 Apr 19 13:40:34 reiserfs .text 0xf8921060 0xf8949697 0xf89497f0 Apr 19 13:40:34 0xeeee599c 0xf8943b3a [reiserfs]flush_commit_list+0xaa (0xf69ab800, 0xeae57b20, 0x1, 0x0) Apr 19 13:40:34 reiserfs .text 0xf8921060 0xf8943a90 0xf8943f60 Apr 19 13:40:34 0xeeee59dc 0xf8943417 [reiserfs]get_list_bitmap+0x77 (0xf69ab800, 0xe2279840, 0x1, 0x2, 0x0) Apr 19 13:40:34 reiserfs .text 0xf8921060 0xf89433a0 0xf8943450 Apr 19 13:40:34 0xeeee5a00 0xf89491a1 [reiserfs]do_journal_end+0x7d1 (0xeeee5a74, 0xf69ab800, 0x1, 0x2, 0xf69ab800) Apr 19 13:40:34 reiserfs .text 0xf8921060 0xf89489d0 0xf89495a0 Apr 19 13:40:34 0xeeee5a64 0xf894753a [reiserfs]do_journal_begin_r+0x14a Apr 19 13:40:34 reiserfs .text 0xf8921060 0xf89473f0 0xf8947670 Apr 19 13:40:34 0xeeee5aa8 0xf89477f2 [reiserfs]journal_begin_Rsmp_c8218e28+0x72 (0xeeee5e54, 0xf69ab800, 0x384, 0x0, 0x2) Apr 19 13:40:35 reiserfs .text 0xf8921060 0xf8947780 0xf8947850 Apr 19 13:40:35 0xeeee5acc 0xf89471f2 [reiserfs]reiserfs_restart_transaction+0x92 (0xeeee5e54, 0x384, 0x2ec7ca2, 0x1, 0x168ec ) Apr 19 13:40:35 [0]more> Apr 19 13:40:36 reiserfs .text 0xf8921060 0xf8947160 0xf8947240 Apr 19 13:40:36 0xeeee5af4 0xf893e9f2 [reiserfs]prepare_for_delete_or_cut+0x622 (0xeeee5e54, 0xd6c0f580, 0xeeee5dd4, 0xeeee5d b4, 0xeeee5bb8) Apr 19 13:40:36 reiserfs .text 0xf8921060 0xf893e3d0 0xf893ebc0 Apr 19 13:40:36 0xeeee5b6c 0xf893fc08 [reiserfs]reiserfs_cut_from_item+0xd8 (0xeeee5e54, 0xeeee5dd4, 0xeeee5db4, 0xd6c0f580, 0x0) Apr 19 13:40:36 reiserfs .text 0xf8921060 0xf893fb30 0xf89401d0 Apr 19 13:40:36 0xeeee5d84 0xf894050e [reiserfs]reiserfs_do_truncate+0x29e Apr 19 13:40:36 reiserfs .text 0xf8921060 0xf8940270 0xf89407d0 Apr 19 13:40:36 0xeeee5e28 0xf893f6ad [reiserfs]reiserfs_delete_object+0x3d (0xeeee5e54, 0xd6c0f580, 0x24, 0xd6c0f5ec, 0xf69a b800) Apr 19 13:40:36 reiserfs .text 0xf8921060 0xf893f670 0xf893f6f0 Apr 19 13:40:37 0xeeee5e44 0xf8929727 [reiserfs]reiserfs_delete_inode+0x107 (0xd6c0f580, 0xffffffff, 0x0) Apr 19 13:40:37 reiserfs .text 0xf8921060 0xf8929620 0xf89297b0 Apr 19 13:40:37 0xeeee5e88 0xc015f12a iput+0x17a Apr 19 13:40:37 kernel .text 0xc0100000 0xc015efb0 0xc015f2e0 Apr 19 13:40:37 0xeeee5ea4 0xc015ca25 d_delete+0xa5 Apr 19 13:40:37 kernel .text 0xc0100000 0xc015c980 0xc015ca40 Apr 19 13:40:37 0xeeee5eb8 0xc0153a85 vfs_unlink+0x185 Apr 19 13:40:37 kernel .text 0xc0100000 0xc0153900 0xc0153be0 Apr 19 13:40:37 0xeeee5ed4 0xf8d23a35 [nfsd]nfsd_unlink+0x125 (0xeeef4c00, 0xeeef4804, 0xffffc000, 0xeeee006c, 0xa) Apr 19 13:40:37 nfsd .text 0xf8d1d060 0xf8d23910 0xf8d23b50 Apr 19 13:40:37 0xeeee5f10 0xf8d2888d [nfsd]nfsd3_proc_remove+0x7d (0xeeef4c00, 0xeeef4a00, 0xeeef4800) Apr 19 13:40:37 nfsd .text 0xf8d1d060 0xf8d28810 0xf8d28920 Apr 19 13:40:38 0xeeee5f48 0xf8d1d64e [nfsd]nfsd_dispatch+0xce (0xeeef4c00, 0xeeee0018, 0xeeee5f8c, 0x94, 0x98) Apr 19 13:40:38 nfsd .text 0xf8d1d060 0xf8d1d580 0xf8d1d765 Apr 19 13:40:38 [0]more> Apr 19 13:40:38 0xeeee5f64 0xf8cfe2cf [sunrpc]svc_process_Rsmp_877fc141+0x45f (0xc28ed260, 0xeeef4c00, 0x7530, 0xb4f38b8, 0xe eee4000) Apr 19 13:40:38 sunrpc .text 0xf8cf6060 0xf8cfde70 0xf8cfe3f5 Apr 19 13:40:39 0xeeee5fa4 0xf8d1d41f [nfsd]nfsd+0x21f Apr 19 13:40:39 nfsd .text 0xf8d1d060 0xf8d1d200 0xf8d1d580 Apr 19 13:40:39 0xeeee5ff4 0xc010741e arch_kernel_thread+0x2e Apr 19 13:40:39 kernel .text 0xc0100000 0xc01073f0 0xc0107430 Thanks for any help you can provide... Jason Keltz ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: reiserfs3 bug? 2.4.32 2006-04-20 13:27 ` reiserfs3 bug? 2.4.32 Jason Keltz @ 2006-04-21 7:47 ` Vladimir V. Saveliev 2006-04-21 12:42 ` Jason Keltz 0 siblings, 1 reply; 4+ messages in thread From: Vladimir V. Saveliev @ 2006-04-21 7:47 UTC (permalink / raw) To: Jason Keltz; +Cc: Chris Mason, reiserfs-list Hello On Thu, 2006-04-20 at 09:27 -0400, Jason Keltz wrote: > On April 5, I had sent an email to the list about a problem that we were > having on our systems that seemed to be reiserfs related. > Unfortunately, I didn't get any response. I'm trying one more time with > other information in the hope that someone might be able to assist us in > solving this problem. > > We're having problems with our file server crashing every 6-30 days. Does it crash or lockup? When a system crashes it usually outputs some information to console. Does it do that in your case? If it does not - please describe system behavour after crash: is it lockuped completely or just some process get blocked? > I > can't remember the last time it stayed up longer than 30 days. I've > recently installed kdb in order to get more debugging information in the > hopes that I might be able to get assistance in solving this problem. > The server crashed for the second time yesterday since I got kdb > installed (18 days since the last time). I've attached some minimal > output from kdb (kreiserfsd process, rpc.mountd, lockd, and then 2 nfsd > processes). Any assistance would be appreciated. > > I e-mailed Neil Brown of NFS server fame, and he confirmed that the > problem was not an NFS server problem, but a reiserfs one. > > Neil gave me this excellent description of the problem: > > The deadlock is happening in flushing out the journal. Any process > that tries to write will need to wait for that journal flush to > complete, which it won't. Hmm, > One nfsd thread is blocked waiting to write. It holds a read lock on > the nfsd exports table. > There is an 'rpc.mountd' process that is trying to get a write lock on > the exports table. It will block until the read lock is released. > The other nfsd threads are trying to get a read lock. That won't > succeed until mountd gets it's write lock and releases it. > > Thus all but one nfsd threads are blocked by mountd which is blocked > by the remaining nfsd which is blocked by kreiserfsd. > > Special things about our nfs server: > > The server runs stock kernel 2.4.32 with kdb patch and reiserfs quota patch. > NFS filesystems being exported are all reiserfs3. > It is a dual processor system running the SMP kernel. > All exported filesystems are on a 3ware raid card. > > The kdb logs: > > Apr 19 13:39:36 0xf67fc000 247 1 0 1 D 0xf67fc370 > kreiserfsd > Apr 19 13:39:36 ESP EIP Function (args) > Apr 19 13:39:36 0xf67fde98 0xc011b144 schedule+0x2b4 (0xc0452e40, 0x0, > 0xf67fc000, 0xc91f8a10, 0xc91f8a10) > Apr 19 13:39:36 kernel .text 0xc0100000 > 0xc011ae90 0xc011b3d0 > Apr 19 13:39:36 0xf67fdee0 0xc014680e __wait_on_buffer+0x6e (0xc91f89c0, > 0x1601, 0xf67fdf30, 0x1, 0xeeee4000) > Apr 19 13:39:36 kernel .text 0xc0100000 > 0xc01467a0 0xc0146840 > Apr 19 13:39:36 0xf67fdf08 0xf8943e79 [reiserfs]flush_commit_list+0x3e9 > (0xf69ab800, 0xeae57b20, 0x0) > Apr 19 13:39:36 reiserfs .text 0xf8921060 > 0xf8943a90 0xf8943f60 > Apr 19 13:39:36 0xf67fdf48 0xf8943a41 [reiserfs]flush_older_commits+0x91 > (0xf69ab800, 0xeae570a0, 0xebc80578, 0xf8ac6000, 0xf > 6a2373c) > Apr 19 13:39:37 reiserfs .text 0xf8921060 > 0xf89439b0 0xf8943a90 > Apr 19 13:39:37 0xf67fdf68 0xf8943b24 [reiserfs]flush_commit_list+0x94 > Apr 19 13:39:37 reiserfs .text 0xf8921060 > 0xf8943a90 0xf8943f60 > Apr 19 13:39:37 0xf67fdfa8 0xf894801d [reiserfs]flush_async_commits+0x3d > (0xf69ab800, 0xd17ebee0, 0xf67fdfd8, 0xf67fdfdc, 0x2 > 0) > Apr 19 13:39:37 reiserfs .text 0xf8921060 > 0xf8947fe0 0xf8948020 > Apr 19 13:39:37 0xf67fdfb8 0xf894652b > [reiserfs]reiserfs_journal_commit_thread+0x1db > Apr 19 13:39:37 reiserfs .text 0xf8921060 > 0xf8946350 0xf89465f0 > Apr 19 13:39:37 0xf67fdff4 0xc010741e arch_kernel_thread+0x2e > Apr 19 13:39:37 kernel .text 0xc0100000 > 0xc01073f0 0xc0107430 > > Apr 19 13:39:47 0xeeffe000 993 1 0 0 S 0xeeffe370 > rpc.mountd > Apr 19 13:39:47 ESP EIP Function (args) > Apr 19 13:39:47 0xeefffe04 0xc011b144 schedule+0x2b4 (0x0, 0xeeffe000, > 0xf8d30164, 0xee98ff88, 0xeeffe000) > Apr 19 13:39:47 kernel .text 0xc0100000 > 0xc011ae90 0xc011b3d0 > Apr 19 13:39:48 0xeefffe4c 0xc011b78f interruptible_sleep_on+0x4f > Apr 19 13:39:48 kernel .text 0xc0100000 > 0xc011b740 0xc011b7d0 > Apr 19 13:39:48 0xeefffe6c 0xf8d24d8a [nfsd]exp_writelock+0x5a > Apr 19 13:39:48 nfsd .text 0xf8d1d060 > 0xf8d24d30 0xf8d24e30 > Apr 19 13:39:48 0xeefffe74 0xf8d244e6 [nfsd]exp_export+0x76 (0xdff64004, > 0xbfffd000, 0x814, 0xc0151015, 0xcf62dc20) > Apr 19 13:39:48 nfsd .text 0xf8d1d060 > 0xf8d24470 0xf8d24880 > Apr 19 13:39:48 0xeefffed4 0xf8d1da4e [nfsd]handle_sys_nfsservctl+0x23e > (0x3, 0xbfffd000, 0x0, 0xeeffe000) > Apr 19 13:39:48 nfsd .text 0xf8d1d060 > 0xf8d1d810 0xf8d1dc90 > Apr 19 13:39:48 0xeeffffac 0xc0160ed6 sys_nfsservctl+0x76 (0x3, > 0xbfffd000, 0x0, 0x420dabf7, 0x2) > Apr 19 13:39:48 kernel .text 0xc0100000 > 0xc0160e60 0xc0160f4b > Apr 19 13:39:48 0xeeffffc4 0xc0108f1f system_call+0x33 > Apr 19 13:39:48 kernel .text 0xc0100000 > 0xc0108eec 0xc0108f24 > > Apr 19 13:40:11 0xeef70000 1021 1 0 1 D 0xeef70370 lockd > Apr 19 13:40:11 ESP EIP Function (args) > Apr 19 13:40:12 0xeef71f60 0xc011b144 schedule+0x2b4 (0x0, 0xeef70000, > 0xeee93f88, 0xf8d30164, 0xeef70000) > Apr 19 13:40:12 kernel .text 0xc0100000 > 0xc011ae90 0xc011b3d0 > Apr 19 13:40:12 0xeef71fa8 0xc011b8af sleep_on+0x4f > Apr 19 13:40:12 kernel .text 0xc0100000 > 0xc011b860 0xc011b8f0 > Apr 19 13:40:12 0xeef71fc8 0xf8d24d0a [nfsd]exp_readlock+0x2a > (0xc28ed2e0, 0xeefa7000, 0x7fffffff, 0xeef70000, 0x4789) > Apr 19 13:40:12 nfsd .text 0xf8d1d060 > 0xf8d24ce0 0xf8d24d30 > Apr 19 13:40:12 0xeef71fcc 0xf8d10149 [lockd]lockd+0x1c9 > Apr 19 13:40:12 lockd .text 0xf8d0e060 > 0xf8d0ff80 0xf8d10260 > Apr 19 13:40:12 0xeef71ff4 0xc010741e arch_kernel_thread+0x2e > Apr 19 13:40:12 kernel .text 0xc0100000 > 0xc01073f0 0xc0107430 > > almost every nfsd was like this: > > Apr 19 13:39:49 0xef00a000 998 1 0 1 D 0xef00a370 nfsd > Apr 19 13:39:49 ESP EIP Function (args) > Apr 19 13:39:49 0xef00bf38 0xc011b144 schedule+0x2b4 (0x0, 0xef00a000, > 0xeed05f88, 0xee997f88, 0x337) > Apr 19 13:39:49 kernel .text 0xc0100000 > 0xc011ae90 0xc011b3d0 > Apr 19 13:39:49 0xef00bf80 0xc011b8af sleep_on+0x4f > Apr 19 13:39:49 kernel .text 0xc0100000 > 0xc011b860 0xc011b8f0 > Apr 19 13:39:49 0xef00bfa0 0xf8d24d0a [nfsd]exp_readlock+0x2a > (0xc28ed260, 0xf002d800, 0x7530, 0xb4fae97, 0xef00a000) > Apr 19 13:39:49 nfsd .text 0xf8d1d060 > 0xf8d24ce0 0xf8d24d30 > Apr 19 13:39:49 0xef00bfa4 0xf8d1d3a8 [nfsd]nfsd+0x1a8 > Apr 19 13:39:49 nfsd .text 0xf8d1d060 > 0xf8d1d200 0xf8d1d580 > Apr 19 13:39:49 0xef00bff4 0xc010741e arch_kernel_thread+0x2e > Apr 19 13:39:49 kernel .text 0xc0100000 > 0xc01073f0 0xc0107430 > and one like this: > > Apr 19 13:40:33 0xeeee4000 1043 1 0 0 D 0xeeee4370 nfsd > Apr 19 13:40:33 ESP EIP Function (args) > Apr 19 13:40:33 0xeeee5920 0xc011b144 schedule+0x2b4 (0x1, 0xeeee4000, > 0xeae57b44, 0xeae57b44, 0x14ca5af6) > Apr 19 13:40:33 kernel .text 0xc0100000 > 0xc011ae90 0xc011b3d0 > Apr 19 13:40:33 0xeeee5968 0xc01079b3 __down+0x83 > Apr 19 13:40:33 kernel .text 0xc0100000 > 0xc0107930 0xc0107a10 > Apr 19 13:40:33 0xeeee598c 0xc0107b5c __down_failed+0x8 (0xf69ab800, > 0xeae57b20, 0xeeee59c4, 0x0, 0xf7fda438) > Apr 19 13:40:33 kernel .text 0xc0100000 > 0xc0107b54 0xc0107b60 > Apr 19 13:40:34 0xeeee599c 0xf894969c [reiserfs].text.lock.journal+0x5 > Apr 19 13:40:34 reiserfs .text 0xf8921060 > 0xf8949697 0xf89497f0 > Apr 19 13:40:34 0xeeee599c 0xf8943b3a [reiserfs]flush_commit_list+0xaa > (0xf69ab800, 0xeae57b20, 0x1, 0x0) > Apr 19 13:40:34 reiserfs .text 0xf8921060 > 0xf8943a90 0xf8943f60 > Apr 19 13:40:34 0xeeee59dc 0xf8943417 [reiserfs]get_list_bitmap+0x77 > (0xf69ab800, 0xe2279840, 0x1, 0x2, 0x0) > Apr 19 13:40:34 reiserfs .text 0xf8921060 > 0xf89433a0 0xf8943450 > Apr 19 13:40:34 0xeeee5a00 0xf89491a1 [reiserfs]do_journal_end+0x7d1 > (0xeeee5a74, 0xf69ab800, 0x1, 0x2, 0xf69ab800) > Apr 19 13:40:34 reiserfs .text 0xf8921060 > 0xf89489d0 0xf89495a0 > Apr 19 13:40:34 0xeeee5a64 0xf894753a [reiserfs]do_journal_begin_r+0x14a > Apr 19 13:40:34 reiserfs .text 0xf8921060 > 0xf89473f0 0xf8947670 > Apr 19 13:40:34 0xeeee5aa8 0xf89477f2 > [reiserfs]journal_begin_Rsmp_c8218e28+0x72 (0xeeee5e54, 0xf69ab800, > 0x384, 0x0, 0x2) > Apr 19 13:40:35 reiserfs .text 0xf8921060 > 0xf8947780 0xf8947850 > Apr 19 13:40:35 0xeeee5acc 0xf89471f2 > [reiserfs]reiserfs_restart_transaction+0x92 (0xeeee5e54, 0x384, > 0x2ec7ca2, 0x1, 0x168ec > ) > Apr 19 13:40:35 [0]more> > Apr 19 13:40:36 reiserfs .text 0xf8921060 > 0xf8947160 0xf8947240 > Apr 19 13:40:36 0xeeee5af4 0xf893e9f2 > [reiserfs]prepare_for_delete_or_cut+0x622 (0xeeee5e54, 0xd6c0f580, > 0xeeee5dd4, 0xeeee5d > b4, 0xeeee5bb8) > Apr 19 13:40:36 reiserfs .text 0xf8921060 > 0xf893e3d0 0xf893ebc0 > Apr 19 13:40:36 0xeeee5b6c 0xf893fc08 > [reiserfs]reiserfs_cut_from_item+0xd8 (0xeeee5e54, 0xeeee5dd4, > 0xeeee5db4, 0xd6c0f580, > 0x0) > Apr 19 13:40:36 reiserfs .text 0xf8921060 > 0xf893fb30 0xf89401d0 > Apr 19 13:40:36 0xeeee5d84 0xf894050e [reiserfs]reiserfs_do_truncate+0x29e > Apr 19 13:40:36 reiserfs .text 0xf8921060 > 0xf8940270 0xf89407d0 > Apr 19 13:40:36 0xeeee5e28 0xf893f6ad > [reiserfs]reiserfs_delete_object+0x3d (0xeeee5e54, 0xd6c0f580, 0x24, > 0xd6c0f5ec, 0xf69a > b800) > Apr 19 13:40:36 reiserfs .text 0xf8921060 > 0xf893f670 0xf893f6f0 > Apr 19 13:40:37 0xeeee5e44 0xf8929727 > [reiserfs]reiserfs_delete_inode+0x107 (0xd6c0f580, 0xffffffff, 0x0) > Apr 19 13:40:37 reiserfs .text 0xf8921060 > 0xf8929620 0xf89297b0 > Apr 19 13:40:37 0xeeee5e88 0xc015f12a iput+0x17a > Apr 19 13:40:37 kernel .text 0xc0100000 > 0xc015efb0 0xc015f2e0 > Apr 19 13:40:37 0xeeee5ea4 0xc015ca25 d_delete+0xa5 > Apr 19 13:40:37 kernel .text 0xc0100000 > 0xc015c980 0xc015ca40 > Apr 19 13:40:37 0xeeee5eb8 0xc0153a85 vfs_unlink+0x185 > Apr 19 13:40:37 kernel .text 0xc0100000 > 0xc0153900 0xc0153be0 > Apr 19 13:40:37 0xeeee5ed4 0xf8d23a35 [nfsd]nfsd_unlink+0x125 > (0xeeef4c00, 0xeeef4804, 0xffffc000, 0xeeee006c, 0xa) > Apr 19 13:40:37 nfsd .text 0xf8d1d060 > 0xf8d23910 0xf8d23b50 > Apr 19 13:40:37 0xeeee5f10 0xf8d2888d [nfsd]nfsd3_proc_remove+0x7d > (0xeeef4c00, 0xeeef4a00, 0xeeef4800) > Apr 19 13:40:37 nfsd .text 0xf8d1d060 > 0xf8d28810 0xf8d28920 > Apr 19 13:40:38 0xeeee5f48 0xf8d1d64e [nfsd]nfsd_dispatch+0xce > (0xeeef4c00, 0xeeee0018, 0xeeee5f8c, 0x94, 0x98) > Apr 19 13:40:38 nfsd .text 0xf8d1d060 > 0xf8d1d580 0xf8d1d765 > Apr 19 13:40:38 [0]more> > Apr 19 13:40:38 0xeeee5f64 0xf8cfe2cf > [sunrpc]svc_process_Rsmp_877fc141+0x45f (0xc28ed260, 0xeeef4c00, 0x7530, > 0xb4f38b8, 0xe > eee4000) > Apr 19 13:40:38 sunrpc .text 0xf8cf6060 > 0xf8cfde70 0xf8cfe3f5 > Apr 19 13:40:39 0xeeee5fa4 0xf8d1d41f [nfsd]nfsd+0x21f > Apr 19 13:40:39 nfsd .text 0xf8d1d060 > 0xf8d1d200 0xf8d1d580 > Apr 19 13:40:39 0xeeee5ff4 0xc010741e arch_kernel_thread+0x2e > Apr 19 13:40:39 kernel .text 0xc0100000 > 0xc01073f0 0xc0107430 > > Thanks for any help you can provide... > > Jason Keltz > ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: reiserfs3 bug? 2.4.32 2006-04-21 7:47 ` Vladimir V. Saveliev @ 2006-04-21 12:42 ` Jason Keltz 0 siblings, 0 replies; 4+ messages in thread From: Jason Keltz @ 2006-04-21 12:42 UTC (permalink / raw) To: Vladimir V. Saveliev; +Cc: Chris Mason, reiserfs-list Hi Vladimir, Vladimir V. Saveliev wrote: > On Thu, 2006-04-20 at 09:27 -0400, Jason Keltz wrote: >> On April 5, I had sent an email to the list about a problem that we were >> having on our systems that seemed to be reiserfs related. >> Unfortunately, I didn't get any response. I'm trying one more time with >> other information in the hope that someone might be able to assist us in >> solving this problem. >> >> We're having problems with our file server crashing every 6-30 days. > > Does it crash or lockup? When a system crashes it usually outputs some > information to console. Does it do that in your case? > If it does not - please describe system behavour after crash: is it > lockuped completely or just some process get blocked? I apologize. My terminology wasn't correct. I guess the system itself is neither crashing nor hanging. NFS activity stops, and all the machines get an "nfs server not responding" message. I can still connect to the file server via SSH, and it doesn't seem to do be doing anything. What I did not try to do was to read and write to ALL reiserfs disks. I imagine that if I get the right disk that is causing the deadlock condition, the system will then hang, but I could be wrong. Any ideas on ways I could add debugging code to the kernel that could help solve the problem? One other minor point that I may have left out. I applied the group of "quota" patches from mason@suse, and not just one of the patches.. 01-reiserfs-free-blocks.diff.gz 05-write_times.diff.gz 02-akpm-b_journal_head.diff.gz 06-reiserfs-quota-28.diff.gz 03-reiserfs-sync_fs-4.diff.gz 07-kinode-10.diff.gz 04-data-logging-40.diff.gz My guess is that the problem is somewhere in these patches since I don't see other people complaining about reiserfs patches, and I imagine that the 2.4 quota patch would be used much less than 2.6 reiserfs with quota. We're working at an upgrade path from our existing systems to 2.6, but it won't happen for a few months, and we really need to solve this problem ... Thanks for any help you can provide.. > >> I >> can't remember the last time it stayed up longer than 30 days. I've >> recently installed kdb in order to get more debugging information in the >> hopes that I might be able to get assistance in solving this problem. >> The server crashed for the second time yesterday since I got kdb >> installed (18 days since the last time). I've attached some minimal >> output from kdb (kreiserfsd process, rpc.mountd, lockd, and then 2 nfsd >> processes). Any assistance would be appreciated. >> >> I e-mailed Neil Brown of NFS server fame, and he confirmed that the >> problem was not an NFS server problem, but a reiserfs one. >> >> Neil gave me this excellent description of the problem: >> >> The deadlock is happening in flushing out the journal. Any process >> that tries to write will need to wait for that journal flush to >> complete, which it won't. > > Hmm, > >> One nfsd thread is blocked waiting to write. It holds a read lock on >> the nfsd exports table. >> There is an 'rpc.mountd' process that is trying to get a write lock on >> the exports table. It will block until the read lock is released. >> The other nfsd threads are trying to get a read lock. That won't >> succeed until mountd gets it's write lock and releases it. >> >> Thus all but one nfsd threads are blocked by mountd which is blocked >> by the remaining nfsd which is blocked by kreiserfsd. >> >> Special things about our nfs server: >> >> The server runs stock kernel 2.4.32 with kdb patch and reiserfs quota patch. >> NFS filesystems being exported are all reiserfs3. >> It is a dual processor system running the SMP kernel. >> All exported filesystems are on a 3ware raid card. >> >> The kdb logs: >> >> Apr 19 13:39:36 0xf67fc000 247 1 0 1 D 0xf67fc370 >> kreiserfsd >> Apr 19 13:39:36 ESP EIP Function (args) >> Apr 19 13:39:36 0xf67fde98 0xc011b144 schedule+0x2b4 (0xc0452e40, 0x0, >> 0xf67fc000, 0xc91f8a10, 0xc91f8a10) >> Apr 19 13:39:36 kernel .text 0xc0100000 >> 0xc011ae90 0xc011b3d0 >> Apr 19 13:39:36 0xf67fdee0 0xc014680e __wait_on_buffer+0x6e (0xc91f89c0, >> 0x1601, 0xf67fdf30, 0x1, 0xeeee4000) >> Apr 19 13:39:36 kernel .text 0xc0100000 >> 0xc01467a0 0xc0146840 >> Apr 19 13:39:36 0xf67fdf08 0xf8943e79 [reiserfs]flush_commit_list+0x3e9 >> (0xf69ab800, 0xeae57b20, 0x0) >> Apr 19 13:39:36 reiserfs .text 0xf8921060 >> 0xf8943a90 0xf8943f60 >> Apr 19 13:39:36 0xf67fdf48 0xf8943a41 [reiserfs]flush_older_commits+0x91 >> (0xf69ab800, 0xeae570a0, 0xebc80578, 0xf8ac6000, 0xf >> 6a2373c) >> Apr 19 13:39:37 reiserfs .text 0xf8921060 >> 0xf89439b0 0xf8943a90 >> Apr 19 13:39:37 0xf67fdf68 0xf8943b24 [reiserfs]flush_commit_list+0x94 >> Apr 19 13:39:37 reiserfs .text 0xf8921060 >> 0xf8943a90 0xf8943f60 >> Apr 19 13:39:37 0xf67fdfa8 0xf894801d [reiserfs]flush_async_commits+0x3d >> (0xf69ab800, 0xd17ebee0, 0xf67fdfd8, 0xf67fdfdc, 0x2 >> 0) >> Apr 19 13:39:37 reiserfs .text 0xf8921060 >> 0xf8947fe0 0xf8948020 >> Apr 19 13:39:37 0xf67fdfb8 0xf894652b >> [reiserfs]reiserfs_journal_commit_thread+0x1db >> Apr 19 13:39:37 reiserfs .text 0xf8921060 >> 0xf8946350 0xf89465f0 >> Apr 19 13:39:37 0xf67fdff4 0xc010741e arch_kernel_thread+0x2e >> Apr 19 13:39:37 kernel .text 0xc0100000 >> 0xc01073f0 0xc0107430 >> >> Apr 19 13:39:47 0xeeffe000 993 1 0 0 S 0xeeffe370 >> rpc.mountd >> Apr 19 13:39:47 ESP EIP Function (args) >> Apr 19 13:39:47 0xeefffe04 0xc011b144 schedule+0x2b4 (0x0, 0xeeffe000, >> 0xf8d30164, 0xee98ff88, 0xeeffe000) >> Apr 19 13:39:47 kernel .text 0xc0100000 >> 0xc011ae90 0xc011b3d0 >> Apr 19 13:39:48 0xeefffe4c 0xc011b78f interruptible_sleep_on+0x4f >> Apr 19 13:39:48 kernel .text 0xc0100000 >> 0xc011b740 0xc011b7d0 >> Apr 19 13:39:48 0xeefffe6c 0xf8d24d8a [nfsd]exp_writelock+0x5a >> Apr 19 13:39:48 nfsd .text 0xf8d1d060 >> 0xf8d24d30 0xf8d24e30 >> Apr 19 13:39:48 0xeefffe74 0xf8d244e6 [nfsd]exp_export+0x76 (0xdff64004, >> 0xbfffd000, 0x814, 0xc0151015, 0xcf62dc20) >> Apr 19 13:39:48 nfsd .text 0xf8d1d060 >> 0xf8d24470 0xf8d24880 >> Apr 19 13:39:48 0xeefffed4 0xf8d1da4e [nfsd]handle_sys_nfsservctl+0x23e >> (0x3, 0xbfffd000, 0x0, 0xeeffe000) >> Apr 19 13:39:48 nfsd .text 0xf8d1d060 >> 0xf8d1d810 0xf8d1dc90 >> Apr 19 13:39:48 0xeeffffac 0xc0160ed6 sys_nfsservctl+0x76 (0x3, >> 0xbfffd000, 0x0, 0x420dabf7, 0x2) >> Apr 19 13:39:48 kernel .text 0xc0100000 >> 0xc0160e60 0xc0160f4b >> Apr 19 13:39:48 0xeeffffc4 0xc0108f1f system_call+0x33 >> Apr 19 13:39:48 kernel .text 0xc0100000 >> 0xc0108eec 0xc0108f24 >> >> Apr 19 13:40:11 0xeef70000 1021 1 0 1 D 0xeef70370 lockd >> Apr 19 13:40:11 ESP EIP Function (args) >> Apr 19 13:40:12 0xeef71f60 0xc011b144 schedule+0x2b4 (0x0, 0xeef70000, >> 0xeee93f88, 0xf8d30164, 0xeef70000) >> Apr 19 13:40:12 kernel .text 0xc0100000 >> 0xc011ae90 0xc011b3d0 >> Apr 19 13:40:12 0xeef71fa8 0xc011b8af sleep_on+0x4f >> Apr 19 13:40:12 kernel .text 0xc0100000 >> 0xc011b860 0xc011b8f0 >> Apr 19 13:40:12 0xeef71fc8 0xf8d24d0a [nfsd]exp_readlock+0x2a >> (0xc28ed2e0, 0xeefa7000, 0x7fffffff, 0xeef70000, 0x4789) >> Apr 19 13:40:12 nfsd .text 0xf8d1d060 >> 0xf8d24ce0 0xf8d24d30 >> Apr 19 13:40:12 0xeef71fcc 0xf8d10149 [lockd]lockd+0x1c9 >> Apr 19 13:40:12 lockd .text 0xf8d0e060 >> 0xf8d0ff80 0xf8d10260 >> Apr 19 13:40:12 0xeef71ff4 0xc010741e arch_kernel_thread+0x2e >> Apr 19 13:40:12 kernel .text 0xc0100000 >> 0xc01073f0 0xc0107430 >> >> almost every nfsd was like this: >> >> Apr 19 13:39:49 0xef00a000 998 1 0 1 D 0xef00a370 nfsd >> Apr 19 13:39:49 ESP EIP Function (args) >> Apr 19 13:39:49 0xef00bf38 0xc011b144 schedule+0x2b4 (0x0, 0xef00a000, >> 0xeed05f88, 0xee997f88, 0x337) >> Apr 19 13:39:49 kernel .text 0xc0100000 >> 0xc011ae90 0xc011b3d0 >> Apr 19 13:39:49 0xef00bf80 0xc011b8af sleep_on+0x4f >> Apr 19 13:39:49 kernel .text 0xc0100000 >> 0xc011b860 0xc011b8f0 >> Apr 19 13:39:49 0xef00bfa0 0xf8d24d0a [nfsd]exp_readlock+0x2a >> (0xc28ed260, 0xf002d800, 0x7530, 0xb4fae97, 0xef00a000) >> Apr 19 13:39:49 nfsd .text 0xf8d1d060 >> 0xf8d24ce0 0xf8d24d30 >> Apr 19 13:39:49 0xef00bfa4 0xf8d1d3a8 [nfsd]nfsd+0x1a8 >> Apr 19 13:39:49 nfsd .text 0xf8d1d060 >> 0xf8d1d200 0xf8d1d580 >> Apr 19 13:39:49 0xef00bff4 0xc010741e arch_kernel_thread+0x2e >> Apr 19 13:39:49 kernel .text 0xc0100000 >> 0xc01073f0 0xc0107430 >> and one like this: >> >> Apr 19 13:40:33 0xeeee4000 1043 1 0 0 D 0xeeee4370 nfsd >> Apr 19 13:40:33 ESP EIP Function (args) >> Apr 19 13:40:33 0xeeee5920 0xc011b144 schedule+0x2b4 (0x1, 0xeeee4000, >> 0xeae57b44, 0xeae57b44, 0x14ca5af6) >> Apr 19 13:40:33 kernel .text 0xc0100000 >> 0xc011ae90 0xc011b3d0 >> Apr 19 13:40:33 0xeeee5968 0xc01079b3 __down+0x83 >> Apr 19 13:40:33 kernel .text 0xc0100000 >> 0xc0107930 0xc0107a10 >> Apr 19 13:40:33 0xeeee598c 0xc0107b5c __down_failed+0x8 (0xf69ab800, >> 0xeae57b20, 0xeeee59c4, 0x0, 0xf7fda438) >> Apr 19 13:40:33 kernel .text 0xc0100000 >> 0xc0107b54 0xc0107b60 >> Apr 19 13:40:34 0xeeee599c 0xf894969c [reiserfs].text.lock.journal+0x5 >> Apr 19 13:40:34 reiserfs .text 0xf8921060 >> 0xf8949697 0xf89497f0 >> Apr 19 13:40:34 0xeeee599c 0xf8943b3a [reiserfs]flush_commit_list+0xaa >> (0xf69ab800, 0xeae57b20, 0x1, 0x0) >> Apr 19 13:40:34 reiserfs .text 0xf8921060 >> 0xf8943a90 0xf8943f60 >> Apr 19 13:40:34 0xeeee59dc 0xf8943417 [reiserfs]get_list_bitmap+0x77 >> (0xf69ab800, 0xe2279840, 0x1, 0x2, 0x0) >> Apr 19 13:40:34 reiserfs .text 0xf8921060 >> 0xf89433a0 0xf8943450 >> Apr 19 13:40:34 0xeeee5a00 0xf89491a1 [reiserfs]do_journal_end+0x7d1 >> (0xeeee5a74, 0xf69ab800, 0x1, 0x2, 0xf69ab800) >> Apr 19 13:40:34 reiserfs .text 0xf8921060 >> 0xf89489d0 0xf89495a0 >> Apr 19 13:40:34 0xeeee5a64 0xf894753a [reiserfs]do_journal_begin_r+0x14a >> Apr 19 13:40:34 reiserfs .text 0xf8921060 >> 0xf89473f0 0xf8947670 >> Apr 19 13:40:34 0xeeee5aa8 0xf89477f2 >> [reiserfs]journal_begin_Rsmp_c8218e28+0x72 (0xeeee5e54, 0xf69ab800, >> 0x384, 0x0, 0x2) >> Apr 19 13:40:35 reiserfs .text 0xf8921060 >> 0xf8947780 0xf8947850 >> Apr 19 13:40:35 0xeeee5acc 0xf89471f2 >> [reiserfs]reiserfs_restart_transaction+0x92 (0xeeee5e54, 0x384, >> 0x2ec7ca2, 0x1, 0x168ec >> ) >> Apr 19 13:40:35 [0]more> >> Apr 19 13:40:36 reiserfs .text 0xf8921060 >> 0xf8947160 0xf8947240 >> Apr 19 13:40:36 0xeeee5af4 0xf893e9f2 >> [reiserfs]prepare_for_delete_or_cut+0x622 (0xeeee5e54, 0xd6c0f580, >> 0xeeee5dd4, 0xeeee5d >> b4, 0xeeee5bb8) >> Apr 19 13:40:36 reiserfs .text 0xf8921060 >> 0xf893e3d0 0xf893ebc0 >> Apr 19 13:40:36 0xeeee5b6c 0xf893fc08 >> [reiserfs]reiserfs_cut_from_item+0xd8 (0xeeee5e54, 0xeeee5dd4, >> 0xeeee5db4, 0xd6c0f580, >> 0x0) >> Apr 19 13:40:36 reiserfs .text 0xf8921060 >> 0xf893fb30 0xf89401d0 >> Apr 19 13:40:36 0xeeee5d84 0xf894050e [reiserfs]reiserfs_do_truncate+0x29e >> Apr 19 13:40:36 reiserfs .text 0xf8921060 >> 0xf8940270 0xf89407d0 >> Apr 19 13:40:36 0xeeee5e28 0xf893f6ad >> [reiserfs]reiserfs_delete_object+0x3d (0xeeee5e54, 0xd6c0f580, 0x24, >> 0xd6c0f5ec, 0xf69a >> b800) >> Apr 19 13:40:36 reiserfs .text 0xf8921060 >> 0xf893f670 0xf893f6f0 >> Apr 19 13:40:37 0xeeee5e44 0xf8929727 >> [reiserfs]reiserfs_delete_inode+0x107 (0xd6c0f580, 0xffffffff, 0x0) >> Apr 19 13:40:37 reiserfs .text 0xf8921060 >> 0xf8929620 0xf89297b0 >> Apr 19 13:40:37 0xeeee5e88 0xc015f12a iput+0x17a >> Apr 19 13:40:37 kernel .text 0xc0100000 >> 0xc015efb0 0xc015f2e0 >> Apr 19 13:40:37 0xeeee5ea4 0xc015ca25 d_delete+0xa5 >> Apr 19 13:40:37 kernel .text 0xc0100000 >> 0xc015c980 0xc015ca40 >> Apr 19 13:40:37 0xeeee5eb8 0xc0153a85 vfs_unlink+0x185 >> Apr 19 13:40:37 kernel .text 0xc0100000 >> 0xc0153900 0xc0153be0 >> Apr 19 13:40:37 0xeeee5ed4 0xf8d23a35 [nfsd]nfsd_unlink+0x125 >> (0xeeef4c00, 0xeeef4804, 0xffffc000, 0xeeee006c, 0xa) >> Apr 19 13:40:37 nfsd .text 0xf8d1d060 >> 0xf8d23910 0xf8d23b50 >> Apr 19 13:40:37 0xeeee5f10 0xf8d2888d [nfsd]nfsd3_proc_remove+0x7d >> (0xeeef4c00, 0xeeef4a00, 0xeeef4800) >> Apr 19 13:40:37 nfsd .text 0xf8d1d060 >> 0xf8d28810 0xf8d28920 >> Apr 19 13:40:38 0xeeee5f48 0xf8d1d64e [nfsd]nfsd_dispatch+0xce >> (0xeeef4c00, 0xeeee0018, 0xeeee5f8c, 0x94, 0x98) >> Apr 19 13:40:38 nfsd .text 0xf8d1d060 >> 0xf8d1d580 0xf8d1d765 >> Apr 19 13:40:38 [0]more> >> Apr 19 13:40:38 0xeeee5f64 0xf8cfe2cf >> [sunrpc]svc_process_Rsmp_877fc141+0x45f (0xc28ed260, 0xeeef4c00, 0x7530, >> 0xb4f38b8, 0xe >> eee4000) >> Apr 19 13:40:38 sunrpc .text 0xf8cf6060 >> 0xf8cfde70 0xf8cfe3f5 >> Apr 19 13:40:39 0xeeee5fa4 0xf8d1d41f [nfsd]nfsd+0x21f >> Apr 19 13:40:39 nfsd .text 0xf8d1d060 >> 0xf8d1d200 0xf8d1d580 >> Apr 19 13:40:39 0xeeee5ff4 0xc010741e arch_kernel_thread+0x2e >> Apr 19 13:40:39 kernel .text 0xc0100000 >> 0xc01073f0 0xc0107430 >> >> Thanks for any help you can provide... >> >> Jason Keltz >> > ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2006-04-21 12:42 UTC | newest] Thread overview: 4+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2006-04-05 14:39 reiserfs3 bug? Jason Keltz 2006-04-20 13:27 ` reiserfs3 bug? 2.4.32 Jason Keltz 2006-04-21 7:47 ` Vladimir V. Saveliev 2006-04-21 12:42 ` Jason Keltz
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.