* linux v3.1 with btrfs-work: oops when deleting files
@ 2011-10-25 4:48 Kai Krakow
2011-10-26 9:35 ` dima
0 siblings, 1 reply; 7+ messages in thread
From: Kai Krakow @ 2011-10-25 4:48 UTC (permalink / raw)
To: linux-btrfs
Hello list!
I'm trying to rm some files, this is what I get in dmesg:
[30975.249519] ------------[ cut here ]------------
[30975.249529] WARNING: at fs/btrfs/extent-tree.c:4588
__btrfs_free_extent+0x3b7/0x7ed()
[30975.249532] Hardware name:
[30975.249535] Modules linked in: af_packet lm90 it87 hwmon_vid hwmon fuse
snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss
snd_mixer_oss reiserfs zram(C) mperf loop emu10k1_gp sidewinder joydev nfs
lockd auth_rpcgss nfs_acl tcp_cubic sunrpc ipv6 snd_usb_audio
snd_usbmidi_lib gspca_sonixj gspca_main videodev usbhid 8250_pnp
i82875p_edac edac_core sr_mod sg analog cdrom firewire_ohci firewire_core
evdev pcspkr parport_pc floppy parport ns558 gameport i2c_i801 ne2k_pci 8390
8250 serial_core snd_mpu401 snd_mpu401_uart intel_agp e1000 crc_itu_t fan
button intel_gtt agpgart processor thermal unix usb_storage hid [last
unloaded: microcode]
[30975.249604] Pid: 12291, comm: rm Tainted: G A C 3.1.0-00057-
gc82b96b-dirty #6
[30975.249607] Call Trace:
[30975.249615] [<c1029497>] ? warn_slowpath_common+0x7c/0x8f
[30975.249620] [<c1142ed2>] ? __btrfs_free_extent+0x3b7/0x7ed
[30975.249624] [<c1142ed2>] ? __btrfs_free_extent+0x3b7/0x7ed
[30975.249628] [<c10294c5>] ? warn_slowpath_null+0x1b/0x1e
[30975.249632] [<c1142ed2>] ? __btrfs_free_extent+0x3b7/0x7ed
[30975.249638] [<c11b7230>] ? rb_erase+0x1ce/0x1f0
[30975.249643] [<c1143b40>] ? run_clustered_refs+0x838/0x87e
[30975.249648] [<c1185a67>] ? btrfs_find_ref_cluster+0xfc/0x13c
[30975.249653] [<c1143c42>] ? btrfs_run_delayed_refs+0xbc/0x13c
[30975.249658] [<c114f901>] ? __btrfs_end_transaction+0x7e/0x1ac
[30975.249663] [<c114fa7f>] ? btrfs_end_transaction+0x11/0x13
[30975.249667] [<c1157cb5>] ? btrfs_evict_inode+0x288/0x29f
[30975.249672] [<c10af24b>] ? evict+0x82/0x11d
[30975.249677] [<c10a8fec>] ? do_unlinkat+0xc9/0x110
[30975.249683] [<c10c3fbb>] ? fsnotify_find_inode_mark+0x17/0x1d
[30975.249688] [<c109d513>] ? filp_close+0x58/0x5f
[30975.249693] [<c1357c93>] ? sysenter_do_call+0x12/0x22
[30975.249697] ---[ end trace 568719abda29cd87 ]---
[30975.249701] btrfs unable to find ref byte nr 1271613026304 parent 0 root
256 owner 445125 offset 2097152
[30975.249713] BUG: unable to handle kernel NULL pointer dereference at
(null)
[30975.249992] IP: [<c11685d8>] map_private_extent_buffer+0x7/0xbc
[30975.250249] *pde = 00000000
[30975.250446] Oops: 0000 [#1] SMP
[30975.250652] Modules linked in: af_packet lm90 it87 hwmon_vid hwmon fuse
snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss
snd_mixer_oss reiserfs zram(C) mperf loop emu10k1_gp sidewinder joydev nfs
lockd auth_rpcgss nfs_acl tcp_cubic sunrpc ipv6 snd_usb_audio
snd_usbmidi_lib gspca_sonixj gspca_main videodev usbhid 8250_pnp
i82875p_edac edac_core sr_mod sg analog cdrom firewire_ohci firewire_core
evdev pcspkr parport_pc floppy parport ns558 gameport i2c_i801 ne2k_pci 8390
8250 serial_core snd_mpu401 snd_mpu401_uart intel_agp e1000 crc_itu_t fan
button intel_gtt agpgart processor thermal unix usb_storage hid [last
unloaded: microcode]
[30975.252788]
[30975.252788] Pid: 12291, comm: rm Tainted: G AWC 3.1.0-00057-
gc82b96b-dirty #6 /8KNXP
[30975.252788] EIP: 0060:[<c11685d8>] EFLAGS: 00210282 CPU: 1
[30975.252788] EIP is at map_private_extent_buffer+0x7/0xbc
[30975.252788] EAX: 00000000 EBX: 00000000 ECX: 00000004 EDX: 0000007a
[30975.252788] ESI: 00000065 EDI: 0000007a EBP: fffffffb ESP: ec80bcf4
[30975.252788] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[30975.252788] Process rm (pid: 12291, ti=ec80a000 task=f1512370
task.ti=ec80a000)
[30975.252788] Stack:
[30975.252788] 335b0009 35373930 3934322e 5d313037 c1020020 c13eac01
00000000 00000065
[30975.252788] 0000007a fffffffb c1161fe2 ec80bd38 ec80bd34 ec80bd30
00000000 0006cac5
[30975.252788] f1f5a7e0 fffffffb 00000000 0006cac5 f1f5a7e0 c1142f61
c13fc992 120af000
[30975.252788] Call Trace:
[30975.252788] [<c1020020>] ? sched_move_task+0x70/0x8e
[30975.252788] [<c1161fe2>] ? btrfs_item_size+0x32/0x63
[30975.252788] [<c1142f61>] ? __btrfs_free_extent+0x446/0x7ed
[30975.252788] [<c11b7230>] ? rb_erase+0x1ce/0x1f0
[30975.252788] [<c1143b40>] ? run_clustered_refs+0x838/0x87e
[30975.252788] [<c1185a67>] ? btrfs_find_ref_cluster+0xfc/0x13c
[30975.252788] [<c1143c42>] ? btrfs_run_delayed_refs+0xbc/0x13c
[30975.252788] [<c114f901>] ? __btrfs_end_transaction+0x7e/0x1ac
[30975.252788] [<c114fa7f>] ? btrfs_end_transaction+0x11/0x13
[30975.252788] [<c1157cb5>] ? btrfs_evict_inode+0x288/0x29f
[30975.252788] [<c10af24b>] ? evict+0x82/0x11d
[30975.252788] [<c10a8fec>] ? do_unlinkat+0xc9/0x110
[30975.252788] [<c10c3fbb>] ? fsnotify_find_inode_mark+0x17/0x1d
[30975.252788] [<c109d513>] ? filp_close+0x58/0x5f
[30975.252788] [<c1357c93>] ? sysenter_do_call+0x12/0x22
[30975.252788] Code: c7 8b 74 24 04 89 e9 f3 a4 89 74 24 04 29 6c 24 1c ff
04 24 31 ff 83 7c 24 1c 00 75 a6 5b 5e 5b 5e 5f 5d c3 55 57 56 53 83 ec 18
<8b> 18 81 e3 ff 0f 00 00 8d 3c 1a 89 fe c1 ee 0c 8d 6c 0f ff c1
[30975.252788] EIP: [<c11685d8>] map_private_extent_buffer+0x7/0xbc SS:ESP
0068:ec80bcf4
[30975.252788] CR2: 0000000000000000
[30975.252788] ------------[ cut here ]------------
[30975.252788] WARNING: at kernel/softirq.c:159
_local_bh_enable_ip+0x2f/0x67()
[30975.252788] Hardware name:
[30975.252788] Modules linked in: af_packet lm90 it87 hwmon_vid hwmon fuse
snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss
snd_mixer_oss reiserfs zram(C) mperf loop emu10k1_gp sidewinder joydev nfs
lockd auth_rpcgss nfs_acl tcp_cubic sunrpc ipv6 snd_usb_audio
snd_usbmidi_lib gspca_sonixj gspca_main videodev usbhid 8250_pnp
i82875p_edac edac_core sr_mod sg analog cdrom firewire_ohci firewire_core
evdev pcspkr parport_pc floppy parport ns558 gameport i2c_i801 ne2k_pci 8390
8250 serial_core snd_mpu401 snd_mpu401_uart intel_agp e1000 crc_itu_t fan
button intel_gtt agpgart processor thermal unix usb_storage hid [last
unloaded: microcode]
[30975.252788] Pid: 12291, comm: rm Tainted: G AWC 3.1.0-00057-
gc82b96b-dirty #6
[30975.252788] Call Trace:
[30975.252788] [<c1029497>] ? warn_slowpath_common+0x7c/0x8f
[30975.252788] [<c102dbbb>] ? _local_bh_enable_ip+0x2f/0x67
[30975.252788] [<c102dbbb>] ? _local_bh_enable_ip+0x2f/0x67
[30975.252788] [<c10294c5>] ? warn_slowpath_null+0x1b/0x1e
[30975.252788] [<c102dbbb>] ? _local_bh_enable_ip+0x2f/0x67
[30975.252788] [<c1230468>] ? cn_netlink_send+0x6d/0x161
[30975.252788] [<c11e20e4>] ? uvesafb_exec+0xef/0x224
[30975.252788] [<c11e2234>] ? uvesafb_prep+0x1b/0x2f
[30975.252788] [<c11e233a>] ? uvesafb_blank+0xf2/0x113
[30975.252788] [<c11db178>] ? fbcon_cursor+0x12b/0x14c
[30975.252788] [<c11d4147>] ? fb_blank+0x2e/0x52
[30975.252788] [<c11dc207>] ? fbcon_blank+0x117/0x23a
[30975.252788] [<c122807a>] ? vt_console_print+0x276/0x284
[30975.252788] [<c1227e04>] ? notify_update+0x22/0x22
[30975.252788] [<c102963a>] ? __call_console_drivers+0x5d/0x69
[30975.252788] [<c1040865>] ? up+0x9/0x2a
[30975.252788] [<c1029b73>] ? console_unlock+0x17c/0x1b2
[30975.252788] [<c10324ac>] ? lock_timer_base+0x19/0x34
[30975.252788] [<c1032954>] ? __mod_timer+0xeb/0xf5
[30975.252788] [<c1032a36>] ? mod_timer+0x76/0x7b
[30975.252788] [<c1229921>] ? do_unblank_screen+0xe7/0x157
[30975.252788] [<c1004b5c>] ? __die+0xe9/0xf2
[30975.252788] [<c11bb79c>] ? bust_spinlocks+0x10/0x2c
[30975.252788] [<c1004b75>] ? oops_end+0x10/0x7d
[30975.252788] [<c1355a84>] ? printk+0x17/0x1b
[30975.252788] [<c101a793>] ? no_context+0x11b/0x123
[30975.252788] [<c101a8fd>] ? bad_area+0x38/0x3d
[30975.252788] [<c101acde>] ? do_page_fault+0x18c/0x2f6
[30975.252788] [<c11b93a9>] ? vsnprintf+0x89/0x263
[30975.252788] [<c1040865>] ? up+0x9/0x2a
[30975.252788] [<c101ab52>] ? vmalloc_sync_all+0xc2/0xc2
[30975.252788] [<c13579df>] ? error_code+0x67/0x6c
[30975.252788] [<c11685d8>] ? map_private_extent_buffer+0x7/0xbc
[30975.252788] [<c1020020>] ? sched_move_task+0x70/0x8e
[30975.252788] [<c1161fe2>] ? btrfs_item_size+0x32/0x63
[30975.252788] [<c1142f61>] ? __btrfs_free_extent+0x446/0x7ed
[30975.252788] [<c11b7230>] ? rb_erase+0x1ce/0x1f0
[30975.252788] [<c1143b40>] ? run_clustered_refs+0x838/0x87e
[30975.252788] [<c1185a67>] ? btrfs_find_ref_cluster+0xfc/0x13c
[30975.252788] [<c1143c42>] ? btrfs_run_delayed_refs+0xbc/0x13c
[30975.252788] [<c114f901>] ? __btrfs_end_transaction+0x7e/0x1ac
[30975.252788] [<c114fa7f>] ? btrfs_end_transaction+0x11/0x13
[30975.252788] [<c1157cb5>] ? btrfs_evict_inode+0x288/0x29f
[30975.252788] [<c10af24b>] ? evict+0x82/0x11d
[30975.252788] [<c10a8fec>] ? do_unlinkat+0xc9/0x110
[30975.252788] [<c10c3fbb>] ? fsnotify_find_inode_mark+0x17/0x1d
[30975.252788] [<c109d513>] ? filp_close+0x58/0x5f
[30975.252788] [<c1357c93>] ? sysenter_do_call+0x12/0x22
[30975.252788] ---[ end trace 568719abda29cd88 ]---
[30975.509259] ---[ end trace 568719abda29cd89 ]---
Kernel is based on linux v3.1 with latest btrfs-work branch from Josef
pulled in:
commit 60f9f544edc06db0833fb35c2ec3ebf37b4bcf1b
Author: Josef Bacik <josef@redhat.com>
Date: Tue Oct 18 13:07:31 2011 -0400
Btrfs: if we have a lot of pinned space, commit the transaction
My kernel version is dirty because it contains the following uncomitted
patch:
diff --git a/fs/btrfs/print-tree.c b/fs/btrfs/print-tree.c
index fb2605d..f38e452 100644
--- a/fs/btrfs/print-tree.c
+++ b/fs/btrfs/print-tree.c
@@ -158,8 +158,7 @@ static void print_extent_ref_v0(struct extent_buffer
*eb, int slot)
void btrfs_print_leaf(struct btrfs_root *root, struct extent_buffer *l)
{
int i;
- u32 type;
- u32 nr = btrfs_header_nritems(l);
+ u32 type, nr;
struct btrfs_item *item;
struct btrfs_root_item *ri;
struct btrfs_dir_item *di;
@@ -172,6 +171,11 @@ void btrfs_print_leaf(struct btrfs_root *root, struct
extent_buffer *l)
struct btrfs_key key;
struct btrfs_key found_key;
+ if (!l)
+ return;
+
+ nr = btrfs_header_nritems(l);
+
printk(KERN_INFO "leaf %llu total ptrs %d free space %d\n",
(unsigned long long)btrfs_header_bytenr(l), nr,
btrfs_leaf_free_space(root, l));
Without this patch, a NULL pointer exception in btrfs_print_lead() usually
hides the other errors.
Thanks for the great work,
Kai
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: linux v3.1 with btrfs-work: oops when deleting files
2011-10-25 4:48 linux v3.1 with btrfs-work: oops when deleting files Kai Krakow
@ 2011-10-26 9:35 ` dima
2011-10-26 10:25 ` Helmut Hullen
2011-10-26 17:40 ` Kai Krakow
0 siblings, 2 replies; 7+ messages in thread
From: dima @ 2011-10-26 9:35 UTC (permalink / raw)
To: Kai Krakow; +Cc: linux-btrfs
On 10/25/2011 01:48 PM, Kai Krakow wrote:
> Hello list!
>
> I'm trying to rm some files, this is what I get in dmesg:
>
> [30975.249519] ------------[ cut here ]------------
> [30975.249529] WARNING: at fs/btrfs/extent-tree.c:4588
> __btrfs_free_extent+0x3b7/0x7ed()
> [30975.249532] Hardware name:
> [30975.249535] Modules linked in: af_packet lm90 it87 hwmon_vid hwmon fuse
> snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss
> snd_mixer_oss reiserfs zram(C) mperf loop emu10k1_gp sidewinder joydev nfs
> lockd auth_rpcgss nfs_acl tcp_cubic sunrpc ipv6 snd_usb_audio
> snd_usbmidi_lib gspca_sonixj gspca_main videodev usbhid 8250_pnp
> i82875p_edac edac_core sr_mod sg analog cdrom firewire_ohci firewire_core
> evdev pcspkr parport_pc floppy parport ns558 gameport i2c_i801 ne2k_pci 8390
> 8250 serial_core snd_mpu401 snd_mpu401_uart intel_agp e1000 crc_itu_t fan
> button intel_gtt agpgart processor thermal unix usb_storage hid [last
> unloaded: microcode]
> [30975.249604] Pid: 12291, comm: rm Tainted: G A C 3.1.0-00057-
> gc82b96b-dirty #6
Hello Kai,
Can you ls the directory where the problem files are located? What would
the the output? I had a very similar problem but on 3.0.x kernel when
several files suddenly got corrupted.
~d
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: linux v3.1 with btrfs-work: oops when deleting files
2011-10-26 9:35 ` dima
@ 2011-10-26 10:25 ` Helmut Hullen
2011-10-27 7:09 ` dima
2011-10-26 17:40 ` Kai Krakow
1 sibling, 1 reply; 7+ messages in thread
From: Helmut Hullen @ 2011-10-26 10:25 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 1776 bytes --]
Hallo, dima,
Du meintest am 26.10.11:
>> I'm trying to rm some files, this is what I get in dmesg:
>>
>> [30975.249519] ------------[ cut here ]------------
>> [30975.249529] WARNING: at fs/btrfs/extent-tree.c:4588
>> __btrfs_free_extent+0x3b7/0x7ed()
[...]
>> [30975.249604] Pid: 12291, comm: rm Tainted: G A C
>> 3.1.0-00057- gc82b96b-dirty #6
> Can you ls the directory where the problem files are located? What
> would the the output? I had a very similar problem but on 3.0.x
> kernel when several files suddenly got corrupted.
This morning I've tried kernel 3.1; you remembder my problems with 1
disk.
dd if=/dev/baddisk of=/dev/zero bs=8M conv=noerror
showed some bad sectors.
hdparm ... --write-sector /dev/baddisk
seems to repair them (I use a loop which not only tests the sector which
is shown via "dd" but also some sectors around this one)
Rebooting the machine with kernel 3.1: I could delete the old entries
which seemed to contain bad sectors. Fine.
Running btrfsck from the "Hugo Mills" git branch: still some errors -
see attachment "btrfsck.txt", especially the last lines; there seems to
be a bug.
Copying some *.mpg files from another place to the btrfs cluster:
suddenly the system hangs, "dmesg" shows similar messages as above (from
Kai Krakow). See second attachment "dmesg-1.txt".
"halt" doesn't work, "reboot" doesn't work, "ctrl alt delete" doesn't
work.
Reboot via power switch.
Again copying: there was (within 1 file) a long pause, but then copying
worked. There's still hope ...
Maybe the pause caused kernel oops #3 and #4 - see attachment "dmesg-
2.txt".
------------
Just to show the only big difference: now I've seen some problem(s) not
related to "rm" but to "cp".
Viele Gruesse!
Helmut
[-- Attachment #2: dmesg-1.txt --]
[-- Type: text/plain, Size: 6238 bytes --]
ata1.00: configured for UDMA/100
ata1.01: configured for UDMA/100
ata1: EH complete
parent transid verify failed on 5340297310208 wanted 57354 found 53683
parent transid verify failed on 5340297310208 wanted 57354 found 53683
parent transid verify failed on 5340297310208 wanted 57354 found 53683
parent transid verify failed on 5340297310208 wanted 57354 found 53683
parent transid verify failed on 5340297310208 wanted 57354 found 53683
BUG: unable to handle kernel NULL pointer dereference at 00000014
IP: [<c1241d76>] btrfs_print_leaf+0x16/0x850
*pdpt = 0000000022784001 *pde = 0000000000000000
Oops: 0000 [#1]
Modules linked in: hisax w83781d hwmon_vid hwmon nf_nat_irc nf_conntrack_irc nf_nat_ftp nf_conntrack_ftp xt_owner ipt_MASQUERADE iptable_nat nf_nat xt_DSCP ipt_LOG xt_multiport xt_recent xt_tcpudp ipt_REJECT iptable_filter iptable_mangle ip_tables xt_iprange nfsd exportfs xt_NOTRACK xt_state ip6t_REJECT ip6table_mangle ip6_tables x_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack ipv6 crc_ccitt isdn slhc 8139too 8139cp mii e1000 aty128fb i2c_i801 piix iTCO_wdt iTCO_vendor_support intel_agp intel_gtt agpgart fuse [last unloaded: hisax]
Pid: 5903, comm: btrfs-endio-wri Not tainted 3.1.0-ODSbig #1 MAXDATA */<P4B>
EIP: 0060:[<c1241d76>] EFLAGS: 00010296 CPU: 0
EIP is at btrfs_print_leaf+0x16/0x850
EAX: d8941c00 EBX: d8941c00 ECX: 00000001 EDX: 00000000
ESI: 00000002 EDI: 00000000 EBP: d3379c54 ESP: d3379be8
DS: 007b ES: 007b FS: 0000 GS: 00e0 SS: 0068
Process btrfs-endio-wri (pid: 5903, ti=d3378000 task=e4a29b80 task.ti=d3378000)
Stack:
00000774 00000774 8ce62000 d8941c00 ffffffff 00000850 85bca000 00000000
d3379c54 00000000 d5015070 00000000 00000000 00000002 00000000 00000000
c122e2f7 000000b0 c122e2f7 00000070 748ce620 a8000007 00001000 00000000
Call Trace:
[<c122e2f7>] ? btrfs_alloc_path+0x17/0x20
[<c122e2f7>] ? btrfs_alloc_path+0x17/0x20
[<c1239d46>] __btrfs_free_extent+0x6e6/0x7c0
[<c10ba9d5>] ? kfree+0xe5/0x110
[<c123ca26>] ? run_clustered_refs+0x106/0x8a0
[<c123cb4f>] run_clustered_refs+0x22f/0x8a0
[<c128ae00>] ? btrfs_find_ref_cluster+0x60/0x1a0
[<c123d259>] btrfs_run_delayed_refs+0x99/0x190
[<c124da86>] __btrfs_end_transaction+0x66/0x1f0
[<c124dc89>] btrfs_end_transaction+0x19/0x20
[<c125445e>] btrfs_finish_ordered_io+0x2ee/0x3c0
[<c1254573>] btrfs_writepage_end_io_hook+0x43/0xa0
[<c126ab41>] end_bio_extent_writepage+0x181/0x1c0
[<c1254530>] ? btrfs_finish_ordered_io+0x3c0/0x3c0
[<c10e6df9>] bio_endio+0x19/0x30
[<c1247835>] end_workqueue_fn+0x125/0x180
[<c17c8d6a>] ? schedule_timeout+0xea/0x1d0
[<c103cf70>] ? cascade+0x80/0x80
[<c1277b21>] worker_loop+0x91/0x350
[<c1028098>] ? __wake_up_common+0x48/0x70
[<c1277a90>] ? btrfs_queue_worker+0x240/0x240
[<c104b594>] kthread+0x74/0x80
[<c104b520>] ? kthread_worker_fn+0xd0/0xd0
[<c17ca93e>] kernel_thread_helper+0x6/0x10
Code: 00 8b 5d f4 8b 75 f8 8b 7d fc 89 ec 5d c3 8d b4 26 00 00 00 00 55 89 e5 83 ec 6c 89 5d f4 89 75 f8 89 7d fc 3e 8d 74 26 00 89 c3 <8b> 42 14 89 d7 e8 70 3a e6 ff 89 fa 8b 40 60 89 45 dc 89 d8 e8
EIP: [<c1241d76>] btrfs_print_leaf+0x16/0x850 SS:ESP 0068:d3379be8
CR2: 0000000000000014
---[ end trace 4f51358d854abcb0 ]---
parent transid verify failed on 5340297310208 wanted 57354 found 53683
parent transid verify failed on 5340297310208 wanted 57354 found 53683
parent transid verify failed on 5340297310208 wanted 57354 found 53683
parent transid verify failed on 5340297310208 wanted 57354 found 53683
parent transid verify failed on 5340297310208 wanted 57354 found 53683
BUG: unable to handle kernel NULL pointer dereference at 00000014
IP: [<c1241d76>] btrfs_print_leaf+0x16/0x850
*pdpt = 0000000036132001 *pde = 0000000000000000
Oops: 0000 [#2]
Modules linked in: hisax w83781d hwmon_vid hwmon nf_nat_irc nf_conntrack_irc nf_nat_ftp nf_conntrack_ftp xt_owner ipt_MASQUERADE iptable_nat nf_nat xt_DSCP ipt_LOG xt_multiport xt_recent xt_tcpudp ipt_REJECT iptable_filter iptable_mangle ip_tables xt_iprange nfsd exportfs xt_NOTRACK xt_state ip6t_REJECT ip6table_mangle ip6_tables x_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack ipv6 crc_ccitt isdn slhc 8139too 8139cp mii e1000 aty128fb i2c_i801 piix iTCO_wdt iTCO_vendor_support intel_agp intel_gtt agpgart fuse [last unloaded: hisax]
Pid: 32278, comm: btrfs-transacti Tainted: G D 3.1.0-ODSbig #1 MAXDATA */<P4B>
EIP: 0060:[<c1241d76>] EFLAGS: 00010296 CPU: 0
EIP is at btrfs_print_leaf+0x16/0x850
EAX: d8941c00 EBX: d8941c00 ECX: 00000001 EDX: 00000000
ESI: 00000002 EDI: 00000000 EBP: c20fbd78 ESP: c20fbd0c
DS: 007b ES: 007b FS: 0000 GS: 00e0 SS: 0068
Process btrfs-transacti (pid: 32278, ti=c20fa000 task=d14494a0 task.ti=c20fa000)
Stack:
00000766 00000774 8a936000 d8941c00 ffffffff 00000850 3f383000 00000000
c20fbd78 00000000 d5015000 00000000 00000000 00000002 00000000 00000000
c122e2f7 000000b0 c122e2f7 00000070 748a9360 a8000007 00001000 00000000
Call Trace:
[<c122e2f7>] ? btrfs_alloc_path+0x17/0x20
[<c122e2f7>] ? btrfs_alloc_path+0x17/0x20
[<c1239d46>] __btrfs_free_extent+0x6e6/0x7c0
[<c10ba9d5>] ? kfree+0xe5/0x110
[<c12ee2fa>] ? rb_erase+0x11a/0x290
[<c123cb4f>] run_clustered_refs+0x22f/0x8a0
[<c128ae00>] ? btrfs_find_ref_cluster+0x60/0x1a0
[<c123d259>] btrfs_run_delayed_refs+0x99/0x190
[<c1268d4b>] ? btrfs_run_ordered_operations+0x15b/0x170
[<c124d318>] btrfs_commit_transaction+0x68/0x740
[<c124dd55>] ? start_transaction+0xc5/0x2a0
[<c124dd05>] ? start_transaction+0x75/0x2a0
[<c104ba40>] ? wake_up_bit+0x30/0x30
[<c124752f>] transaction_kthread+0x1cf/0x1e0
[<c1247360>] ? btrfs_congested_fn+0x90/0x90
[<c1247360>] ? btrfs_congested_fn+0x90/0x90
[<c104b594>] kthread+0x74/0x80
[<c104b520>] ? kthread_worker_fn+0xd0/0xd0
[<c17ca93e>] kernel_thread_helper+0x6/0x10
Code: 00 8b 5d f4 8b 75 f8 8b 7d fc 89 ec 5d c3 8d b4 26 00 00 00 00 55 89 e5 83 ec 6c 89 5d f4 89 75 f8 89 7d fc 3e 8d 74 26 00 89 c3 <8b> 42 14 89 d7 e8 70 3a e6 ff 89 fa 8b 40 60 89 45 dc 89 d8 e8
EIP: [<c1241d76>] btrfs_print_leaf+0x16/0x850 SS:ESP 0068:c20fbd0c
CR2: 0000000000000014
---[ end trace 4f51358d854abcb1 ]---
[-- Attachment #3: dmesg-2.txt --]
[-- Type: text/plain, Size: 5523 bytes --]
Oops: 0000 [#3]
Modules linked in: hisax w83781d hwmon_vid hwmon nf_nat_irc nf_conntrack_irc nf_nat_ftp nf_conntrack_ftp xt_owner ipt_MASQUERADE iptable_nat nf_nat xt_DSCP ipt_LOG xt_multiport xt_recent xt_tcpudp ipt_REJECT iptable_filter iptable_mangle ip_tables xt_iprange nfsd exportfs xt_NOTRACK xt_state ip6t_REJECT ip6table_mangle ip6_tables x_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack ipv6 crc_ccitt isdn slhc 8139too 8139cp mii e1000 aty128fb i2c_i801 piix iTCO_wdt iTCO_vendor_support intel_agp intel_gtt agpgart fuse [last unloaded: hisax]
Pid: 18455, comm: smbd Tainted: G D 3.1.0-ODSbig #1 MAXDATA */<P4B>
EIP: 0060:[<c1241d76>] EFLAGS: 00210286 CPU: 0
EIP is at btrfs_print_leaf+0x16/0x850
EAX: db4de800 EBX: db4de800 ECX: 00000001 EDX: 00000000
ESI: 00000007 EDI: 00000000 EBP: f6e13cfc ESP: f6e13c90
DS: 007b ES: 007b FS: 0000 GS: 00e0 SS: 0068
Process smbd (pid: 18455, ti=f6e12000 task=d0001810 task.ti=f6e12000)
Stack:
000007a7 000007a7 a53f5000 db4de800 ffffffff 00000850 a53f4000 00000000
f6e13cfc 00000000 dcff8000 00000000 00000000 00000007 00000000 00000000
c122e2f7 000000b0 c122e2f7 00000070 a7a53f50 a8000007 00001000 00000000
Call Trace:
[<c122e2f7>] ? btrfs_alloc_path+0x17/0x20
[<c122e2f7>] ? btrfs_alloc_path+0x17/0x20
[<c1239d46>] __btrfs_free_extent+0x6e6/0x7c0
[<c10ba9d5>] ? kfree+0xe5/0x110
[<c12ee3c9>] ? rb_erase+0x1e9/0x290
[<c123cb4f>] run_clustered_refs+0x22f/0x8a0
[<c122d861>] ? leaf_space_used+0x81/0xb0
[<c128ae00>] ? btrfs_find_ref_cluster+0x60/0x1a0
[<c123d259>] btrfs_run_delayed_refs+0x99/0x190
[<c124da86>] __btrfs_end_transaction+0x66/0x1f0
[<c1253a84>] ? btrfs_orphan_del+0xb4/0xe0
[<c124dc89>] btrfs_end_transaction+0x19/0x20
[<c125896f>] btrfs_evict_inode+0x20f/0x270
[<c10d2d2f>] evict+0x6f/0x140
[<c10d2edb>] iput+0xdb/0x1a0
[<c10ca934>] do_unlinkat+0x114/0x160
[<c10cb565>] sys_unlink+0x15/0x20
[<c17c9dfc>] syscall_call+0x7/0xb
[<c17c0000>] ? mvs_64xx_init+0x233/0x6cc
Code: 00 8b 5d f4 8b 75 f8 8b 7d fc 89 ec 5d c3 8d b4 26 00 00 00 00 55 89 e5 83 ec 6c 89 5d f4 89 75 f8 89 7d fc 3e 8d 74 26 00 89 c3 <8b> 42 14 89 d7 e8 70 3a e6 ff 89 fa 8b 40 60 89 45 dc 89 d8 e8
EIP: [<c1241d76>] btrfs_print_leaf+0x16/0x850 SS:ESP 0068:f6e13c90
CR2: 0000000000000014
---[ end trace e823394b87ecbfba ]---
parent transid verify failed on 5340297986048 wanted 57354 found 42286
parent transid verify failed on 5340297986048 wanted 57354 found 42286
parent transid verify failed on 5340297986048 wanted 57354 found 42286
parent transid verify failed on 5340297986048 wanted 57354 found 42286
parent transid verify failed on 5340297986048 wanted 57354 found 42286
BUG: unable to handle kernel NULL pointer dereference at 00000014
IP: [<c1241d76>] btrfs_print_leaf+0x16/0x850
*pdpt = 0000000018af8001 *pde = 0000000000000000
Oops: 0000 [#4]
Modules linked in: hisax w83781d hwmon_vid hwmon nf_nat_irc nf_conntrack_irc nf_nat_ftp nf_conntrack_ftp xt_owner ipt_MASQUERADE iptable_nat nf_nat xt_DSCP ipt_LOG xt_multiport xt_recent xt_tcpudp ipt_REJECT iptable_filter iptable_mangle ip_tables xt_iprange nfsd exportfs xt_NOTRACK xt_state ip6t_REJECT ip6table_mangle ip6_tables x_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack ipv6 crc_ccitt isdn slhc 8139too 8139cp mii e1000 aty128fb i2c_i801 piix iTCO_wdt iTCO_vendor_support intel_agp intel_gtt agpgart fuse [last unloaded: hisax]
Pid: 18484, comm: smbd Tainted: G D 3.1.0-ODSbig #1 MAXDATA */<P4B>
EIP: 0060:[<c1241d76>] EFLAGS: 00210286 CPU: 0
EIP is at btrfs_print_leaf+0x16/0x850
EAX: db4de800 EBX: db4de800 ECX: 00000001 EDX: 00000000
ESI: 00000007 EDI: 00000000 EBP: f6e13cfc ESP: f6e13c90
DS: 007b ES: 007b FS: 0000 GS: 00e0 SS: 0068
Process smbd (pid: 18484, ti=f6e12000 task=db4c46e0 task.ti=f6e12000)
Stack:
00000000 000007a7 a53fa000 db4de800 ffffffff 0000e231 f6e13ce4 00000000
00000000 00000000 dcff8150 00000000 00000000 00000007 00200282 00000000
c122e2f7 000000b0 c122e2f7 00000070 a7a53fa0 a8000007 00001000 00000000
Call Trace:
[<c122e2f7>] ? btrfs_alloc_path+0x17/0x20
[<c122e2f7>] ? btrfs_alloc_path+0x17/0x20
[<c1239d46>] __btrfs_free_extent+0x6e6/0x7c0
[<c1249a2a>] ? btrfs_mark_buffer_dirty+0x7a/0xe0
[<c123cb4f>] run_clustered_refs+0x22f/0x8a0
[<c122e700>] ? btrfs_free_path+0x20/0x30
[<c1257590>] ? btrfs_truncate_inode_items+0x1e0/0x7d0
[<c128ae00>] ? btrfs_find_ref_cluster+0x60/0x1a0
[<c123d259>] btrfs_run_delayed_refs+0x99/0x190
[<c124da86>] __btrfs_end_transaction+0x66/0x1f0
[<c124dc89>] btrfs_end_transaction+0x19/0x20
[<c125890e>] btrfs_evict_inode+0x1ae/0x270
[<c10d2d2f>] evict+0x6f/0x140
[<c10d2edb>] iput+0xdb/0x1a0
[<c10ca934>] do_unlinkat+0x114/0x160
[<c10cb565>] sys_unlink+0x15/0x20
[<c17c9dfc>] syscall_call+0x7/0xb
[<c17c0000>] ? mvs_64xx_init+0x233/0x6cc
Code: 00 8b 5d f4 8b 75 f8 8b 7d fc 89 ec 5d c3 8d b4 26 00 00 00 00 55 89 e5 83 ec 6c 89 5d f4 89 75 f8 89 7d fc 3e 8d 74 26 00 89 c3 <8b> 42 14 89 d7 e8 70 3a e6 ff 89 fa 8b 40 60 89 45 dc 89 d8 e8
EIP: [<c1241d76>] btrfs_print_leaf+0x16/0x850 SS:ESP 0068:f6e13c90
CR2: 0000000000000014
---[ end trace e823394b87ecbfbb ]---
parent transid verify failed on 5340297986048 wanted 57354 found 42286
parent transid verify failed on 5340297986048 wanted 57354 found 42286
parent transid verify failed on
[-- Attachment #4: btrfsck.txt --]
[-- Type: text/plain, Size: 3759 bytes --]
parent transid verify failed on 21094400 wanted 57401 found 47987
parent transid verify failed on 8520776257536 wanted 57347 found 55978
parent transid verify failed on 8520219820032 wanted 57343 found 55967
parent transid verify failed on 8520776273920 wanted 57347 found 55978
parent transid verify failed on 8520776278016 wanted 57347 found 55978
parent transid verify failed on 8520219832320 wanted 57343 found 55967
parent transid verify failed on 8520219836416 wanted 57343 found 55967
parent transid verify failed on 8520219844608 wanted 57343 found 55967
leaf parent key incorrect 5340295811072
bad block 5340295811072
leaf parent key incorrect 5340295839744
bad block 5340295839744
parent transid verify failed on 8520218161152 wanted 57343 found 55967
parent transid verify failed on 8520218169344 wanted 57343 found 55967
leaf parent key incorrect 8520218673152
bad block 8520218673152
parent transid verify failed on 8520218828800 wanted 57343 found 55967
leaf parent key incorrect 8520775094272
bad block 8520775094272
leaf parent key incorrect 8520777793536
bad block 8520777793536
parent transid verify failed on 5340298461184 wanted 57354 found 53683
parent transid verify failed on 8520777793536 wanted 57362 found 57240
parent transid verify failed on 8520777793536 wanted 57362 found 57240
parent transid verify failed on 5340295811072 wanted 57354 found 42286
parent transid verify failed on 5340295811072 wanted 57354 found 42286
parent transid verify failed on 5340295839744 wanted 57354 found 55983
parent transid verify failed on 5340295839744 wanted 57354 found 55983
parent transid verify failed on 5340295843840 wanted 57354 found 55983
parent transid verify failed on 8520777674752 wanted 57362 found 57240
parent transid verify failed on 8520491139072 wanted 57359 found 55974
parent transid verify failed on 8520218828800 wanted 57343 found 55967
leaf parent key incorrect 8520220254208
bad block 8520220254208
leaf parent key incorrect 8520486453248
bad block 8520486453248
leaf parent key incorrect 8520489959424
bad block 8520489959424
leaf parent key incorrect 8520490131456
bad block 8520490131456
bad block 8520773492736
Extent back ref already exists for 4478103113728 parent 0 root 2
Extent back ref already exists for 8520469598208 parent 0 root 2
Extent back ref already exists for 8519937826816 parent 0 root 2
Extent back ref already exists for 8519937830912 parent 0 root 2
Extent back ref already exists for 8521415835648 parent 0 root 2
leaf parent key incorrect 8520774410240
bad block 8520774410240
parent transid verify failed on 8520775753728 wanted 57347 found 55978
leaf parent key incorrect 8521083891712
bad block 8521083891712
leaf parent key incorrect 8521314054144
bad block 8521314054144
parent transid verify failed on 5340296237056 wanted 57354 found 42286
parent transid verify failed on 8519939354624 wanted 57355 found 55960
parent transid verify failed on 8519939211264 wanted 57355 found 57322
parent transid verify failed on 5340295839744 wanted 57354 found 55983
leaf parent key incorrect 21094400
bad block 21094400
leaf parent key incorrect 4478083121152
bad block 4478083121152
bad block 4478083129344
Extent back ref already exists for 4478083346432 parent 0 root 7
Extent back ref already exists for 7039505383424 parent 0 root 7
[...]
Extent back ref already exists for 4776267329536 parent 0 root 7
Extent back ref already exists for 4776268664832 parent 0 root 7
Extent back ref already exists for 4776268619776 parent 0 root 7
leaf parent key incorrect 4478084227072
bad block 4478084227072
warning, start mismatch 4782494081024 4782475907072
btrfsck: btrfsck.c:2611: run_next_block: Assertion `!(ret)' failed.
sh: line 57: 8083 Aborted btrfsck /dev/sdb1
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: linux v3.1 with btrfs-work: oops when deleting files
2011-10-26 9:35 ` dima
2011-10-26 10:25 ` Helmut Hullen
@ 2011-10-26 17:40 ` Kai Krakow
2011-10-26 18:52 ` Helmut Hullen
2011-10-27 1:15 ` dima
1 sibling, 2 replies; 7+ messages in thread
From: Kai Krakow @ 2011-10-26 17:40 UTC (permalink / raw)
To: linux-btrfs
Hello!
2011/10/26 dima <dolenin@parallels.com>:
>> I'm trying to rm some files, this is what I get in dmesg:
[snip]
>
> Can you ls the directory where the problem files are located? What would the
> the output? I had a very similar problem but on 3.0.x kernel when several
> files suddenly got corrupted.
I can run "find -type f" for directories I suspect corrupted files in,
and I see errors in dmesg if it happens to contain bad files. But no
oopses and the system remains stable. If I mount the filesystem
read-only I can even read these files without oopses, this way I
produced an rsync backup to my original partition I created the btrfs
subvolume from.
But since I do not have enough capacity to hold both my original
backup (home files only, the system is already outdated) and the
complete btrfs system (combined multiple partitions from an lvm setup
into one btrfs volume with subvolumes) I cannot just recreate the same
file system from scratch. Looks like I have to buy another hard disk
(going to become expensive soon anyway) and rsync over to a fresh
btrfs. This includes a added bonus of trying out btrfs striping.
Awful, that's going to take about 30 hours again... :-(
I'd like to contribute fixing these bugs but I'm starting to need a
fully running system again with all my data.
All I can say: My files got suddenly corrupted after trying to "cp
--reflink" a VMware player disk image (several gigabytes). I wanted to
do this to easily go back to the original state of the VM but instead
my system started thrashing the hard disk and after some minutes the
cp command returned with an out-of-memory error. From then on, more
and more programs just froze (Chromium being the first) and I could no
longer open terminals ("failed to start bash" after a couple of
seconds). I was able to see oops traces in dmesg in one terminal but
wasn't able to capture it somewhere because I could not log in with
ssh, neither I could create files (all processes froze in disk state,
not killable). So I did a hard reboot with REISUB.
Regards,
Kai
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: linux v3.1 with btrfs-work: oops when deleting files
2011-10-26 17:40 ` Kai Krakow
@ 2011-10-26 18:52 ` Helmut Hullen
2011-10-27 1:15 ` dima
1 sibling, 0 replies; 7+ messages in thread
From: Helmut Hullen @ 2011-10-26 18:52 UTC (permalink / raw)
To: linux-btrfs
Hallo, Kai,
Du meintest am 26.10.11:
> I can run "find -type f" for directories I suspect corrupted files
> in, and I see errors in dmesg if it happens to contain bad files. But
> no oopses and the system remains stable. If I mount the filesystem
> read-only I can even read these files without oopses,
Ok - may be I need this hint too sometimes in the next weeks.
> But since I do not have enough capacity to hold both my original
> backup (home files only, the system is already outdated) and the
> complete btrfs system (combined multiple partitions from an lvm setup
> into one btrfs volume with subvolumes) I cannot just recreate the
> same file system from scratch. Looks like I have to buy another hard
> disk (going to become expensive soon anyway) and rsync over to a
> fresh btrfs.
Same problem here. I fear that I have to copy my btrfs cluster (more
than 4 TByte on 3 disks) to another 3-disk cluster - last week that
would have been 200 Euro, now it may be 300 Euro.
I don't dare the way
add a 4th 2-TByte disk
balance
remove the bad disk
(that was the major reason for me using btrfs - adding bigger disks,
removing smaller disks ...)
Viele Gruesse!
Helmut
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: linux v3.1 with btrfs-work: oops when deleting files
2011-10-26 17:40 ` Kai Krakow
2011-10-26 18:52 ` Helmut Hullen
@ 2011-10-27 1:15 ` dima
1 sibling, 0 replies; 7+ messages in thread
From: dima @ 2011-10-27 1:15 UTC (permalink / raw)
To: Kai Krakow; +Cc: linux-btrfs
On 10/27/2011 02:40 AM, Kai Krakow wrote:
>>> I'm trying to rm some files, this is what I get in dmesg:
> [snip]
>>
>> Can you ls the directory where the problem files are located? What would the
>> the output? I had a very similar problem but on 3.0.x kernel when several
>> files suddenly got corrupted.
>
> I can run "find -type f" for directories I suspect corrupted files in,
> and I see errors in dmesg if it happens to contain bad files. But no
> oopses and the system remains stable. If I mount the filesystem
> read-only I can even read these files without oopses, this way I
> produced an rsync backup to my original partition I created the btrfs
> subvolume from.
I see. Just to share my experience. I think I had a different case
because I could not even read some 20-30 files (photos) on my /home
subvolume and any attempt to access them via rm or ls would give me
kernel oopses, though the system would not crash with 3.1 kernel.
Unless I tried to access the corrupted files the filesystem was stable
and my / subvolume was fine. So I backed up everything not-corrupted (I
did have the backups for the corrupted files anyway) and created my
/home subvolume anew. I deleted the old subvolume and most of the space
was reclaimed.
But btrfsck would still give me errors about inability to access some
inode (
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: linux v3.1 with btrfs-work: oops when deleting files
2011-10-26 10:25 ` Helmut Hullen
@ 2011-10-27 7:09 ` dima
0 siblings, 0 replies; 7+ messages in thread
From: dima @ 2011-10-27 7:09 UTC (permalink / raw)
To: helmut; +Cc: Helmut Hullen, linux-btrfs
On 10/26/2011 07:25 PM, Helmut Hullen wrote:
> Hallo, dima,
>
> Du meintest am 26.10.11:
>
>>> I'm trying to rm some files, this is what I get in dmesg:
>>>
>>> [30975.249519] ------------[ cut here ]------------
>>> [30975.249529] WARNING: at fs/btrfs/extent-tree.c:4588
>>> __btrfs_free_extent+0x3b7/0x7ed()
>
> [...]
>
>>> [30975.249604] Pid: 12291, comm: rm Tainted: G A C
>>> 3.1.0-00057- gc82b96b-dirty #6
>
>> Can you ls the directory where the problem files are located? What
>> would the the output? I had a very similar problem but on 3.0.x
>> kernel when several files suddenly got corrupted.
>
> This morning I've tried kernel 3.1; you remembder my problems with 1
> disk.
>
> dd if=/dev/baddisk of=/dev/zero bs=8M conv=noerror
>
> showed some bad sectors.
>
> hdparm ... --write-sector /dev/baddisk
>
> seems to repair them (I use a loop which not only tests the sector which
> is shown via "dd" but also some sectors around this one)
>
> Rebooting the machine with kernel 3.1: I could delete the old entries
> which seemed to contain bad sectors. Fine.
>
> Running btrfsck from the "Hugo Mills" git branch: still some errors -
> see attachment "btrfsck.txt", especially the last lines; there seems to
> be a bug.
>
> Copying some *.mpg files from another place to the btrfs cluster:
> suddenly the system hangs, "dmesg" shows similar messages as above (from
> Kai Krakow). See second attachment "dmesg-1.txt".
> "halt" doesn't work, "reboot" doesn't work, "ctrl alt delete" doesn't
> work.
>
> Reboot via power switch.
>
> Again copying: there was (within 1 file) a long pause, but then copying
> worked. There's still hope ...
> Maybe the pause caused kernel oops #3 and #4 - see attachment "dmesg-
> 2.txt".
>
> ------------
>
> Just to show the only big difference: now I've seen some problem(s) not
> related to "rm" but to "cp".
>
> Viele Gruesse!
> Helmut
Hi Helmut,
For me any command I tried to use on the corrupted files would give me
the errors in dmesg, like in your case. And I could not get rid of the
files in any way. So eventually I had to recreate the subvolume.
I checked my disk at that time, but it did not show any bad sector
errors, so I concluded it is the FS problem.
In your case it may be just the errors caused by bad sectors on disk.
But perhaps recreating subvolume would be a good step to find out what
is wrong.
~d
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2011-10-27 7:09 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-10-25 4:48 linux v3.1 with btrfs-work: oops when deleting files Kai Krakow
2011-10-26 9:35 ` dima
2011-10-26 10:25 ` Helmut Hullen
2011-10-27 7:09 ` dima
2011-10-26 17:40 ` Kai Krakow
2011-10-26 18:52 ` Helmut Hullen
2011-10-27 1:15 ` dima
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).