* Re: Networking-related crash?
[not found] <608c44bf0912090546s446bf973ne408e99661fdc56f@mail.gmail.com>
@ 2009-12-09 15:11 ` Avi Kivity
2009-12-09 20:36 ` Adam Huffman
2009-12-09 20:54 ` Eric Dumazet
0 siblings, 2 replies; 8+ messages in thread
From: Avi Kivity @ 2009-12-09 15:11 UTC (permalink / raw)
To: Adam Huffman; +Cc: kvm, netdev
On 12/09/2009 03:46 PM, Adam Huffman wrote:
> I've been seeing lots of crashes on a new Dell Precision T7500,
> running the KVM in Fedora 12. Finally managed to capture an Oops,
> which is shown below (hand-transcribed):
>
> BUG: unable to handle kernel paging request at 0000000000200200
> IP: [<ffffffff8139aab7>] destroy_conntrack+0x82/0x11f
> PGD 332d0e067 PUD 33453c067 PMD 0
> Oops: 0002 [#1] SMP
> last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map
> CPU 4
> Modules linked in: tun bridge stp llc sunrpc ipt_MASQUERADE
> iptable_nat nf_nat ipt_LOG xt_physdev ip6t_REJECT nf_conntrack_ipv6
> ip6table_filter ip6
> _tables ipv6 dm_multipath kvm_intel kvm uinput snd_hda_codec_analog
> nouveau snd_hda_intel snd_hda_codec ttm drm_kms_helper snd_hwdep
> snd_seq drm sn
> d_seq_device snd_pcm firewire_ohci i2c_i801 snd_timer ppdev
> firewire_core snd i2c_algo_bit iTCO_wdt crc_itu_t parport_pc i2c_core
> soundcore parport
> iTCO_vendor_support tg3 snd_page_alloc shpchp dcdbas wmi mptsas
> mptscsih mptbase scsi_transport_sas megaraid_sas [last_unloaded:
> speedstep_lib]
> Pid: 1759, comm: qemu-kvm Not tainted 2.6.31.6-162.fc12.x86_64 #1
> Precision WorkStation T7500
> RIP: 0010:[<ffffffff8139aab7>] [<ffffffff8139aab7>]
> destroy_conntrack+0x82/0x11f
> RSP: 0018:ffffc90000803bf0 EFLAGS: 00010202
> RAX: 0000000080000001 RBX: ffffffff816fb1a0 RCX: 000000000000752f
> RDX: 0000000000200200 RSI: 0000000000000011 RDI: ffffffff816fb1a0
> RBP: ffffc90000803c00 R08: ffff880336699438 R09: 0000000000aaa5e0
> R10: 00000002f54189d5 R11: 0000000000000001 R12: ffffffff819a92e0
> R13: ffffffffa029adcc R14: 0000000000000000 R15: ffff880632866c38
> FS: 00007fdd34b17710(0000) GS:ffffc90000800000(0000) knlGS:0000000000000000
> CS: 0010 DS: 002B ES: 002B CR0: 0000000080050033
> CR2: 0000000000200200 CR3: 00000003349c0000 CR4: 00000000000026e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process qemu-kvm (pid: 1759, threadinfo ffff88062e9e8000, task ffff880634945e00)
> Stack:
> ffff880632866c00 ffff880634640c30 ffffc90000803c10 ffffffff813989c2
> <0> ffffc90000803c30 ffffffff81374092 ffffc90000803c30 ffff880632866c00
> <0> ffffc90000803c50 ffffffff81373dd3 0000000200000000 ffff880632866c00
> Call Trace:
> <IRQ>
> [<ffffffff813989c2>] nf_conntrack_destroy+0x1b/0x1d
> [<ffffffff81374092>] skb_release_head_state+0x95/0xd7
> [<ffffffff81373dd3>] __kfree_skb+0x16/0x81
> [<ffffffff81373ed7>] kfree_skb+0x6a/0x72
> [<ffffffffa029adcc>] ip6_mc_input+0x220/0x230 [ipv6]
> [<ffffffffa029a3d1>] ip6_rcv_finish+0x27/0x2b [ipv6]
> [<ffffffffa029a763>] ipv6_rcv+0x38e/0x3e5 [ipv6]
> [<ffffffff8137bd91>] netif_receive_skb+0x402/0x427
> [<ffffffff8137bf1b>] napi_skb_finish+0x29/0x3d
> [<ffffffff8137c37a>] napi_gro_receive+0x2f/0x34
> [<ffffffffa0084fad>] tg3_poll+0x6c6/0x8c3 [tg3]
> [<ffffffff8137c4b0>] net_rx_action+0xaf/0x1c9
> [<ffffffff81379cfe>] ? list-add_tail+0x15/0x17
> [<ffffffff81057614>] __do_softirq+0xdd/0x1ad
> [<ffffffff81026936>] ? apic_write+0x16/0x18
> [<ffffffff81012eac>] call_softirq+0x1c/0x30
> [<ffffffff810143fb>] do_softirq+0x47/0x8d
> [<ffffffff81057326>] irq_exit+0x44/0x86
> [<ffffffff8141ecd5>] do_IRQ+0xa5/0xbc
> [<ffffffff810126d3>] ret_from_intr+0x0/0x11
> <EOI>
> [<ffffffffa02437bb>] ? kvm_arch_vcpu_ioctl_run+0x84b/0xb34 [kvm]
> [<ffffffffa02437aa>] ? kvm_arch_vcpu_ioctl_run+0x83a/0xb34 [kvm]
> [<ffffffffa02395e3>] ? kvm_vcpu_ioctl+0xfd/0x556 [kvm]
> [<ffffffff81108adc>] ? vfs_ioctl+0x22/0x87
> [<ffffffff81109038>] ? do_vfs_ioctl+0x47b/0x4c1
> [<ffffffff811090d4>] ? sys_ioctl+0x56/0x79
> [<ffffffff81012093>] ? stub_clone+0x13/0x20
> [<ffffffff81011cf2>] ? system_call_fastpath+0x16/0x1b
> Code: c7 00 a6 9a 81 e8 23 04 08 00 48 89 df e8 68 29 00 00 f6 43 78
> 08 75 24 48 8b 53 10 48 85 d2 75 04 0f 0b eb fe 48 8b 43 08 a8 01<48>
> 89 02 7
> 5 04 48 89 50 08 48 c7 43 10 00 02 20 00 65 8b 14 25
> RIP [<ffffffff8139aab7>] destroy_conntrack+0x82/0x11f
> RSP<ffffc90000803bf0>
> CR2: 0000000000200200
>
Looks unrelated to kvm - softirq happened to trigger during a kvm
ioctl. Fault looks like list poison. Copying netdev.
--
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Networking-related crash?
2009-12-09 15:11 ` Networking-related crash? Avi Kivity
@ 2009-12-09 20:36 ` Adam Huffman
2009-12-09 20:54 ` Eric Dumazet
1 sibling, 0 replies; 8+ messages in thread
From: Adam Huffman @ 2009-12-09 20:36 UTC (permalink / raw)
To: Avi Kivity; +Cc: kvm, netdev
On Wed, Dec 9, 2009 at 3:11 PM, Avi Kivity <avi@redhat.com> wrote:
> On 12/09/2009 03:46 PM, Adam Huffman wrote:
>>
>> I've been seeing lots of crashes on a new Dell Precision T7500,
>> running the KVM in Fedora 12. Finally managed to capture an Oops,
>> which is shown below (hand-transcribed):
>>
>> BUG: unable to handle kernel paging request at 0000000000200200
>> IP: [<ffffffff8139aab7>] destroy_conntrack+0x82/0x11f
>> PGD 332d0e067 PUD 33453c067 PMD 0
>> Oops: 0002 [#1] SMP
>> last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map
>> CPU 4
>> Modules linked in: tun bridge stp llc sunrpc ipt_MASQUERADE
>> iptable_nat nf_nat ipt_LOG xt_physdev ip6t_REJECT nf_conntrack_ipv6
>> ip6table_filter ip6
>> _tables ipv6 dm_multipath kvm_intel kvm uinput snd_hda_codec_analog
>> nouveau snd_hda_intel snd_hda_codec ttm drm_kms_helper snd_hwdep
>> snd_seq drm sn
>> d_seq_device snd_pcm firewire_ohci i2c_i801 snd_timer ppdev
>> firewire_core snd i2c_algo_bit iTCO_wdt crc_itu_t parport_pc i2c_core
>> soundcore parport
>> iTCO_vendor_support tg3 snd_page_alloc shpchp dcdbas wmi mptsas
>> mptscsih mptbase scsi_transport_sas megaraid_sas [last_unloaded:
>> speedstep_lib]
>> Pid: 1759, comm: qemu-kvm Not tainted 2.6.31.6-162.fc12.x86_64 #1
>> Precision WorkStation T7500
>> RIP: 0010:[<ffffffff8139aab7>] [<ffffffff8139aab7>]
>> destroy_conntrack+0x82/0x11f
>> RSP: 0018:ffffc90000803bf0 EFLAGS: 00010202
>> RAX: 0000000080000001 RBX: ffffffff816fb1a0 RCX: 000000000000752f
>> RDX: 0000000000200200 RSI: 0000000000000011 RDI: ffffffff816fb1a0
>> RBP: ffffc90000803c00 R08: ffff880336699438 R09: 0000000000aaa5e0
>> R10: 00000002f54189d5 R11: 0000000000000001 R12: ffffffff819a92e0
>> R13: ffffffffa029adcc R14: 0000000000000000 R15: ffff880632866c38
>> FS: 00007fdd34b17710(0000) GS:ffffc90000800000(0000)
>> knlGS:0000000000000000
>> CS: 0010 DS: 002B ES: 002B CR0: 0000000080050033
>> CR2: 0000000000200200 CR3: 00000003349c0000 CR4: 00000000000026e0
>> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Process qemu-kvm (pid: 1759, threadinfo ffff88062e9e8000, task
>> ffff880634945e00)
>> Stack:
>> ffff880632866c00 ffff880634640c30 ffffc90000803c10 ffffffff813989c2
>> <0> ffffc90000803c30 ffffffff81374092 ffffc90000803c30 ffff880632866c00
>> <0> ffffc90000803c50 ffffffff81373dd3 0000000200000000 ffff880632866c00
>> Call Trace:
>> <IRQ>
>> [<ffffffff813989c2>] nf_conntrack_destroy+0x1b/0x1d
>> [<ffffffff81374092>] skb_release_head_state+0x95/0xd7
>> [<ffffffff81373dd3>] __kfree_skb+0x16/0x81
>> [<ffffffff81373ed7>] kfree_skb+0x6a/0x72
>> [<ffffffffa029adcc>] ip6_mc_input+0x220/0x230 [ipv6]
>> [<ffffffffa029a3d1>] ip6_rcv_finish+0x27/0x2b [ipv6]
>> [<ffffffffa029a763>] ipv6_rcv+0x38e/0x3e5 [ipv6]
>> [<ffffffff8137bd91>] netif_receive_skb+0x402/0x427
>> [<ffffffff8137bf1b>] napi_skb_finish+0x29/0x3d
>> [<ffffffff8137c37a>] napi_gro_receive+0x2f/0x34
>> [<ffffffffa0084fad>] tg3_poll+0x6c6/0x8c3 [tg3]
>> [<ffffffff8137c4b0>] net_rx_action+0xaf/0x1c9
>> [<ffffffff81379cfe>] ? list-add_tail+0x15/0x17
>> [<ffffffff81057614>] __do_softirq+0xdd/0x1ad
>> [<ffffffff81026936>] ? apic_write+0x16/0x18
>> [<ffffffff81012eac>] call_softirq+0x1c/0x30
>> [<ffffffff810143fb>] do_softirq+0x47/0x8d
>> [<ffffffff81057326>] irq_exit+0x44/0x86
>> [<ffffffff8141ecd5>] do_IRQ+0xa5/0xbc
>> [<ffffffff810126d3>] ret_from_intr+0x0/0x11
>> <EOI>
>> [<ffffffffa02437bb>] ? kvm_arch_vcpu_ioctl_run+0x84b/0xb34 [kvm]
>> [<ffffffffa02437aa>] ? kvm_arch_vcpu_ioctl_run+0x83a/0xb34 [kvm]
>> [<ffffffffa02395e3>] ? kvm_vcpu_ioctl+0xfd/0x556 [kvm]
>> [<ffffffff81108adc>] ? vfs_ioctl+0x22/0x87
>> [<ffffffff81109038>] ? do_vfs_ioctl+0x47b/0x4c1
>> [<ffffffff811090d4>] ? sys_ioctl+0x56/0x79
>> [<ffffffff81012093>] ? stub_clone+0x13/0x20
>> [<ffffffff81011cf2>] ? system_call_fastpath+0x16/0x1b
>> Code: c7 00 a6 9a 81 e8 23 04 08 00 48 89 df e8 68 29 00 00 f6 43 78
>> 08 75 24 48 8b 53 10 48 85 d2 75 04 0f 0b eb fe 48 8b 43 08 a8 01<48>
>> 89 02 7
>> 5 04 48 89 50 08 48 c7 43 10 00 02 20 00 65 8b 14 25
>> RIP [<ffffffff8139aab7>] destroy_conntrack+0x82/0x11f
>> RSP<ffffc90000803bf0>
>> CR2: 0000000000200200
>>
>
> Looks unrelated to kvm - softirq happened to trigger during a kvm ioctl.
> Fault looks like list poison. Copying netdev.
>
Disabling VT-d support in the BIOS seems to have stopped the crashes.
At least it's been running without crashing for several hours now,
while it would only last minutes before.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Networking-related crash?
2009-12-09 15:11 ` Networking-related crash? Avi Kivity
2009-12-09 20:36 ` Adam Huffman
@ 2009-12-09 20:54 ` Eric Dumazet
2009-12-10 11:01 ` Patrick McHardy
1 sibling, 1 reply; 8+ messages in thread
From: Eric Dumazet @ 2009-12-09 20:54 UTC (permalink / raw)
To: Avi Kivity; +Cc: Adam Huffman, kvm, netdev, Patrick McHardy
Le 09/12/2009 16:11, Avi Kivity a écrit :
> On 12/09/2009 03:46 PM, Adam Huffman wrote:
>> I've been seeing lots of crashes on a new Dell Precision T7500,
>> running the KVM in Fedora 12. Finally managed to capture an Oops,
>> which is shown below (hand-transcribed):
>>
>> BUG: unable to handle kernel paging request at 0000000000200200
>> IP: [<ffffffff8139aab7>] destroy_conntrack+0x82/0x11f
>> PGD 332d0e067 PUD 33453c067 PMD 0
>> Oops: 0002 [#1] SMP
>> last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map
>> CPU 4
>> Modules linked in: tun bridge stp llc sunrpc ipt_MASQUERADE
>> iptable_nat nf_nat ipt_LOG xt_physdev ip6t_REJECT nf_conntrack_ipv6
>> ip6table_filter ip6
>> _tables ipv6 dm_multipath kvm_intel kvm uinput snd_hda_codec_analog
>> nouveau snd_hda_intel snd_hda_codec ttm drm_kms_helper snd_hwdep
>> snd_seq drm sn
>> d_seq_device snd_pcm firewire_ohci i2c_i801 snd_timer ppdev
>> firewire_core snd i2c_algo_bit iTCO_wdt crc_itu_t parport_pc i2c_core
>> soundcore parport
>> iTCO_vendor_support tg3 snd_page_alloc shpchp dcdbas wmi mptsas
>> mptscsih mptbase scsi_transport_sas megaraid_sas [last_unloaded:
>> speedstep_lib]
>> Pid: 1759, comm: qemu-kvm Not tainted 2.6.31.6-162.fc12.x86_64 #1
>> Precision WorkStation T7500
>> RIP: 0010:[<ffffffff8139aab7>] [<ffffffff8139aab7>]
>> destroy_conntrack+0x82/0x11f
>> RSP: 0018:ffffc90000803bf0 EFLAGS: 00010202
>> RAX: 0000000080000001 RBX: ffffffff816fb1a0 RCX: 000000000000752f
>> RDX: 0000000000200200 RSI: 0000000000000011 RDI: ffffffff816fb1a0
>> RBP: ffffc90000803c00 R08: ffff880336699438 R09: 0000000000aaa5e0
>> R10: 00000002f54189d5 R11: 0000000000000001 R12: ffffffff819a92e0
>> R13: ffffffffa029adcc R14: 0000000000000000 R15: ffff880632866c38
>> FS: 00007fdd34b17710(0000) GS:ffffc90000800000(0000)
>> knlGS:0000000000000000
>> CS: 0010 DS: 002B ES: 002B CR0: 0000000080050033
>> CR2: 0000000000200200 CR3: 00000003349c0000 CR4: 00000000000026e0
>> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> Process qemu-kvm (pid: 1759, threadinfo ffff88062e9e8000, task
>> ffff880634945e00)
>> Stack:
>> ffff880632866c00 ffff880634640c30 ffffc90000803c10 ffffffff813989c2
>> <0> ffffc90000803c30 ffffffff81374092 ffffc90000803c30 ffff880632866c00
>> <0> ffffc90000803c50 ffffffff81373dd3 0000000200000000 ffff880632866c00
>> Call Trace:
>> <IRQ>
>> [<ffffffff813989c2>] nf_conntrack_destroy+0x1b/0x1d
>> [<ffffffff81374092>] skb_release_head_state+0x95/0xd7
>> [<ffffffff81373dd3>] __kfree_skb+0x16/0x81
>> [<ffffffff81373ed7>] kfree_skb+0x6a/0x72
>> [<ffffffffa029adcc>] ip6_mc_input+0x220/0x230 [ipv6]
>> [<ffffffffa029a3d1>] ip6_rcv_finish+0x27/0x2b [ipv6]
>> [<ffffffffa029a763>] ipv6_rcv+0x38e/0x3e5 [ipv6]
>> [<ffffffff8137bd91>] netif_receive_skb+0x402/0x427
>> [<ffffffff8137bf1b>] napi_skb_finish+0x29/0x3d
>> [<ffffffff8137c37a>] napi_gro_receive+0x2f/0x34
>> [<ffffffffa0084fad>] tg3_poll+0x6c6/0x8c3 [tg3]
>> [<ffffffff8137c4b0>] net_rx_action+0xaf/0x1c9
>> [<ffffffff81379cfe>] ? list-add_tail+0x15/0x17
>> [<ffffffff81057614>] __do_softirq+0xdd/0x1ad
>> [<ffffffff81026936>] ? apic_write+0x16/0x18
>> [<ffffffff81012eac>] call_softirq+0x1c/0x30
>> [<ffffffff810143fb>] do_softirq+0x47/0x8d
>> [<ffffffff81057326>] irq_exit+0x44/0x86
>> [<ffffffff8141ecd5>] do_IRQ+0xa5/0xbc
>> [<ffffffff810126d3>] ret_from_intr+0x0/0x11
>> <EOI>
>> [<ffffffffa02437bb>] ? kvm_arch_vcpu_ioctl_run+0x84b/0xb34 [kvm]
>> [<ffffffffa02437aa>] ? kvm_arch_vcpu_ioctl_run+0x83a/0xb34 [kvm]
>> [<ffffffffa02395e3>] ? kvm_vcpu_ioctl+0xfd/0x556 [kvm]
>> [<ffffffff81108adc>] ? vfs_ioctl+0x22/0x87
>> [<ffffffff81109038>] ? do_vfs_ioctl+0x47b/0x4c1
>> [<ffffffff811090d4>] ? sys_ioctl+0x56/0x79
>> [<ffffffff81012093>] ? stub_clone+0x13/0x20
>> [<ffffffff81011cf2>] ? system_call_fastpath+0x16/0x1b
>> Code: c7 00 a6 9a 81 e8 23 04 08 00 48 89 df e8 68 29 00 00 f6 43 78
>> 08 75 24 48 8b 53 10 48 85 d2 75 04 0f 0b eb fe 48 8b 43 08 a8 01<48>
>> 89 02 7
>> 5 04 48 89 50 08 48 c7 43 10 00 02 20 00 65 8b 14 25
>> RIP [<ffffffff8139aab7>] destroy_conntrack+0x82/0x11f
>> RSP<ffffc90000803bf0>
>> CR2: 0000000000200200
>>
>
> Looks unrelated to kvm - softirq happened to trigger during a kvm
> ioctl. Fault looks like list poison. Copying netdev.
>
crash in :
48 8b 43 08 mov 0x8(%rbx),%rax
a8 01 test $0x1,%al
48 89 02 mov %rax,(%rdx) << HERE >> RDX=0x200200 (LIST_POISON2)
75 04 jne 1f
48 89 50 08 mov %rdx,0x8(%rax)
1: 48 c7 43 10 00 02 20 movq $0x200200,0x10(%rbx)
if (!nf_ct_is_confirmed(ct)) {
BUG_ON(hlist_nulls_unhashed(&ct->tuplehash[IP_CT_DIR_ORIGINAL].hnnode));
hlist_nulls_del_rcu(&ct->tuplehash[IP_CT_DIR_ORIGINAL].hnnode); << HERE >>
}
NF_CT_STAT_INC(net, delete);
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Networking-related crash?
2009-12-09 20:54 ` Eric Dumazet
@ 2009-12-10 11:01 ` Patrick McHardy
2009-12-10 15:18 ` Adam Huffman
0 siblings, 1 reply; 8+ messages in thread
From: Patrick McHardy @ 2009-12-10 11:01 UTC (permalink / raw)
To: Eric Dumazet
Cc: Avi Kivity, Adam Huffman, kvm, netdev,
Netfilter Development Mailinglist
Eric Dumazet wrote:
> Le 09/12/2009 16:11, Avi Kivity a écrit :
>> On 12/09/2009 03:46 PM, Adam Huffman wrote:
>>> I've been seeing lots of crashes on a new Dell Precision T7500,
>>> running the KVM in Fedora 12. Finally managed to capture an Oops,
>>> which is shown below (hand-transcribed):
>>>
>>> BUG: unable to handle kernel paging request at 0000000000200200
>>> IP: [<ffffffff8139aab7>] destroy_conntrack+0x82/0x11f
>>> PGD 332d0e067 PUD 33453c067 PMD 0
>>> RIP: 0010:[<ffffffff8139aab7>] [<ffffffff8139aab7>]
>>> destroy_conntrack+0x82/0x11f
>>> RSP: 0018:ffffc90000803bf0 EFLAGS: 00010202
>>> RAX: 0000000080000001 RBX: ffffffff816fb1a0 RCX: 000000000000752f
>>> RDX: 0000000000200200 RSI: 0000000000000011 RDI: ffffffff816fb1a0
>>> RBP: ffffc90000803c00 R08: ffff880336699438 R09: 0000000000aaa5e0
>>> R10: 00000002f54189d5 R11: 0000000000000001 R12: ffffffff819a92e0
>>> R13: ffffffffa029adcc R14: 0000000000000000 R15: ffff880632866c38
>>> FS: 00007fdd34b17710(0000) GS:ffffc90000800000(0000)
>>> knlGS:0000000000000000
>>> CS: 0010 DS: 002B ES: 002B CR0: 0000000080050033
>>> CR2: 0000000000200200 CR3: 00000003349c0000 CR4: 00000000000026e0
>>> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>>> Process qemu-kvm (pid: 1759, threadinfo ffff88062e9e8000, task
>>> ffff880634945e00)
>>> Stack:
>>> ffff880632866c00 ffff880634640c30 ffffc90000803c10 ffffffff813989c2
>>> <0> ffffc90000803c30 ffffffff81374092 ffffc90000803c30 ffff880632866c00
>>> <0> ffffc90000803c50 ffffffff81373dd3 0000000200000000 ffff880632866c00
>>> Call Trace:
>>> <IRQ>
>>> [<ffffffff813989c2>] nf_conntrack_destroy+0x1b/0x1d
>>> [<ffffffff81374092>] skb_release_head_state+0x95/0xd7
>>> [<ffffffff81373dd3>] __kfree_skb+0x16/0x81
>>> [<ffffffff81373ed7>] kfree_skb+0x6a/0x72
>>> [<ffffffffa029adcc>] ip6_mc_input+0x220/0x230 [ipv6]
>>> [<ffffffffa029a3d1>] ip6_rcv_finish+0x27/0x2b [ipv6]
>>> [<ffffffffa029a763>] ipv6_rcv+0x38e/0x3e5 [ipv6]
>>> [<ffffffff8137bd91>] netif_receive_skb+0x402/0x427
>>> ...
>>>
> crash in :
> 48 8b 43 08 mov 0x8(%rbx),%rax
> a8 01 test $0x1,%al
> 48 89 02 mov %rax,(%rdx) << HERE >> RDX=0x200200 (LIST_POISON2)
> 75 04 jne 1f
> 48 89 50 08 mov %rdx,0x8(%rax)
> 1: 48 c7 43 10 00 02 20 movq $0x200200,0x10(%rbx)
>
> if (!nf_ct_is_confirmed(ct)) {
> BUG_ON(hlist_nulls_unhashed(&ct->tuplehash[IP_CT_DIR_ORIGINAL].hnnode));
> hlist_nulls_del_rcu(&ct->tuplehash[IP_CT_DIR_ORIGINAL].hnnode); << HERE >>
> }
> NF_CT_STAT_INC(net, delete);
I can't spot the problem. Adam, please send me your .config file.
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Networking-related crash?
2009-12-10 11:01 ` Patrick McHardy
@ 2009-12-10 15:18 ` Adam Huffman
2009-12-14 13:16 ` Patrick McHardy
0 siblings, 1 reply; 8+ messages in thread
From: Adam Huffman @ 2009-12-10 15:18 UTC (permalink / raw)
To: Patrick McHardy
Cc: Eric Dumazet, Avi Kivity, kvm, netdev,
Netfilter Development Mailinglist
[-- Attachment #1: Type: text/plain, Size: 3210 bytes --]
On Thu, Dec 10, 2009 at 11:01 AM, Patrick McHardy <kaber@trash.net> wrote:
> Eric Dumazet wrote:
>> Le 09/12/2009 16:11, Avi Kivity a écrit :
>>> On 12/09/2009 03:46 PM, Adam Huffman wrote:
>>>> I've been seeing lots of crashes on a new Dell Precision T7500,
>>>> running the KVM in Fedora 12. Finally managed to capture an Oops,
>>>> which is shown below (hand-transcribed):
>>>>
>>>> BUG: unable to handle kernel paging request at 0000000000200200
>>>> IP: [<ffffffff8139aab7>] destroy_conntrack+0x82/0x11f
>>>> PGD 332d0e067 PUD 33453c067 PMD 0
>>>> RIP: 0010:[<ffffffff8139aab7>] [<ffffffff8139aab7>]
>>>> destroy_conntrack+0x82/0x11f
>>>> RSP: 0018:ffffc90000803bf0 EFLAGS: 00010202
>>>> RAX: 0000000080000001 RBX: ffffffff816fb1a0 RCX: 000000000000752f
>>>> RDX: 0000000000200200 RSI: 0000000000000011 RDI: ffffffff816fb1a0
>>>> RBP: ffffc90000803c00 R08: ffff880336699438 R09: 0000000000aaa5e0
>>>> R10: 00000002f54189d5 R11: 0000000000000001 R12: ffffffff819a92e0
>>>> R13: ffffffffa029adcc R14: 0000000000000000 R15: ffff880632866c38
>>>> FS: 00007fdd34b17710(0000) GS:ffffc90000800000(0000)
>>>> knlGS:0000000000000000
>>>> CS: 0010 DS: 002B ES: 002B CR0: 0000000080050033
>>>> CR2: 0000000000200200 CR3: 00000003349c0000 CR4: 00000000000026e0
>>>> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>>> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>>>> Process qemu-kvm (pid: 1759, threadinfo ffff88062e9e8000, task
>>>> ffff880634945e00)
>>>> Stack:
>>>> ffff880632866c00 ffff880634640c30 ffffc90000803c10 ffffffff813989c2
>>>> <0> ffffc90000803c30 ffffffff81374092 ffffc90000803c30 ffff880632866c00
>>>> <0> ffffc90000803c50 ffffffff81373dd3 0000000200000000 ffff880632866c00
>>>> Call Trace:
>>>> <IRQ>
>>>> [<ffffffff813989c2>] nf_conntrack_destroy+0x1b/0x1d
>>>> [<ffffffff81374092>] skb_release_head_state+0x95/0xd7
>>>> [<ffffffff81373dd3>] __kfree_skb+0x16/0x81
>>>> [<ffffffff81373ed7>] kfree_skb+0x6a/0x72
>>>> [<ffffffffa029adcc>] ip6_mc_input+0x220/0x230 [ipv6]
>>>> [<ffffffffa029a3d1>] ip6_rcv_finish+0x27/0x2b [ipv6]
>>>> [<ffffffffa029a763>] ipv6_rcv+0x38e/0x3e5 [ipv6]
>>>> [<ffffffff8137bd91>] netif_receive_skb+0x402/0x427
>>>> ...
>>>>
>> crash in :
>> 48 8b 43 08 mov 0x8(%rbx),%rax
>> a8 01 test $0x1,%al
>> 48 89 02 mov %rax,(%rdx) << HERE >> RDX=0x200200 (LIST_POISON2)
>> 75 04 jne 1f
>> 48 89 50 08 mov %rdx,0x8(%rax)
>> 1: 48 c7 43 10 00 02 20 movq $0x200200,0x10(%rbx)
>>
>> if (!nf_ct_is_confirmed(ct)) {
>> BUG_ON(hlist_nulls_unhashed(&ct->tuplehash[IP_CT_DIR_ORIGINAL].hnnode));
>> hlist_nulls_del_rcu(&ct->tuplehash[IP_CT_DIR_ORIGINAL].hnnode); << HERE >>
>> }
>> NF_CT_STAT_INC(net, delete);
>
>
> I can't spot the problem. Adam, please send me your .config file.
>
>
It's the standard Fedora .config, which is attached.
As I stated in another message, the oops seems related to VT-d. With
that disabled, the machine has been stable for nearly a day now.
[-- Attachment #2: .config --]
[-- Type: application/x-config, Size: 97986 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Networking-related crash?
2009-12-10 15:18 ` Adam Huffman
@ 2009-12-14 13:16 ` Patrick McHardy
2009-12-14 13:46 ` Adam Huffman
2010-01-05 16:57 ` Adam Huffman
0 siblings, 2 replies; 8+ messages in thread
From: Patrick McHardy @ 2009-12-14 13:16 UTC (permalink / raw)
To: Adam Huffman
Cc: Eric Dumazet, Avi Kivity, kvm, netdev,
Netfilter Development Mailinglist
Adam Huffman wrote:
> On Thu, Dec 10, 2009 at 11:01 AM, Patrick McHardy <kaber@trash.net> wrote:
>> Eric Dumazet wrote:
>>> Le 09/12/2009 16:11, Avi Kivity a écrit :
>>>> On 12/09/2009 03:46 PM, Adam Huffman wrote:
>>>>> I've been seeing lots of crashes on a new Dell Precision T7500,
>>>>> running the KVM in Fedora 12. Finally managed to capture an Oops,
>>>>> which is shown below (hand-transcribed):
>>>>>
>>>>> BUG: unable to handle kernel paging request at 0000000000200200
>>>>> IP: [<ffffffff8139aab7>] destroy_conntrack+0x82/0x11f
>>>>> PGD 332d0e067 PUD 33453c067 PMD 0
>>>>> RIP: 0010:[<ffffffff8139aab7>] [<ffffffff8139aab7>]
>>>>> destroy_conntrack+0x82/0x11f
>>>>> RSP: 0018:ffffc90000803bf0 EFLAGS: 00010202
>>>>> RAX: 0000000080000001 RBX: ffffffff816fb1a0 RCX: 000000000000752f
>>>>> RDX: 0000000000200200 RSI: 0000000000000011 RDI: ffffffff816fb1a0
>>>>> RBP: ffffc90000803c00 R08: ffff880336699438 R09: 0000000000aaa5e0
>>>>> R10: 00000002f54189d5 R11: 0000000000000001 R12: ffffffff819a92e0
>>>>> R13: ffffffffa029adcc R14: 0000000000000000 R15: ffff880632866c38
>>>>> FS: 00007fdd34b17710(0000) GS:ffffc90000800000(0000)
>>>>> knlGS:0000000000000000
>>>>> CS: 0010 DS: 002B ES: 002B CR0: 0000000080050033
>>>>> CR2: 0000000000200200 CR3: 00000003349c0000 CR4: 00000000000026e0
>>>>> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>>>> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>>>>> Process qemu-kvm (pid: 1759, threadinfo ffff88062e9e8000, task
>>>>> ffff880634945e00)
>>>>> Stack:
>>>>> ffff880632866c00 ffff880634640c30 ffffc90000803c10 ffffffff813989c2
>>>>> <0> ffffc90000803c30 ffffffff81374092 ffffc90000803c30 ffff880632866c00
>>>>> <0> ffffc90000803c50 ffffffff81373dd3 0000000200000000 ffff880632866c00
>>>>> Call Trace:
>>>>> <IRQ>
>>>>> [<ffffffff813989c2>] nf_conntrack_destroy+0x1b/0x1d
>>>>> [<ffffffff81374092>] skb_release_head_state+0x95/0xd7
>>>>> [<ffffffff81373dd3>] __kfree_skb+0x16/0x81
>>>>> [<ffffffff81373ed7>] kfree_skb+0x6a/0x72
>>>>> [<ffffffffa029adcc>] ip6_mc_input+0x220/0x230 [ipv6]
>>>>> [<ffffffffa029a3d1>] ip6_rcv_finish+0x27/0x2b [ipv6]
>>>>> [<ffffffffa029a763>] ipv6_rcv+0x38e/0x3e5 [ipv6]
>>>>> [<ffffffff8137bd91>] netif_receive_skb+0x402/0x427
>>>>> ...
>>>>>
>>> crash in :
>>> 48 8b 43 08 mov 0x8(%rbx),%rax
>>> a8 01 test $0x1,%al
>>> 48 89 02 mov %rax,(%rdx) << HERE >> RDX=0x200200 (LIST_POISON2)
>>> 75 04 jne 1f
>>> 48 89 50 08 mov %rdx,0x8(%rax)
>>> 1: 48 c7 43 10 00 02 20 movq $0x200200,0x10(%rbx)
>>>
>>> if (!nf_ct_is_confirmed(ct)) {
>>> BUG_ON(hlist_nulls_unhashed(&ct->tuplehash[IP_CT_DIR_ORIGINAL].hnnode));
>>> hlist_nulls_del_rcu(&ct->tuplehash[IP_CT_DIR_ORIGINAL].hnnode); << HERE >>
>>> }
>>> NF_CT_STAT_INC(net, delete);
>>
>> I can't spot the problem. Adam, please send me your .config file.
>>
>>
>
> It's the standard Fedora .config, which is attached.
>
> As I stated in another message, the oops seems related to VT-d. With
> that disabled, the machine has been stable for nearly a day now.
That probably only affects the timing of some race. Please also
send me the IPv6 ruleset used on that machine. Thanks.
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Networking-related crash?
2009-12-14 13:16 ` Patrick McHardy
@ 2009-12-14 13:46 ` Adam Huffman
2010-01-05 16:57 ` Adam Huffman
1 sibling, 0 replies; 8+ messages in thread
From: Adam Huffman @ 2009-12-14 13:46 UTC (permalink / raw)
To: Patrick McHardy
Cc: Eric Dumazet, Avi Kivity, kvm, netdev,
Netfilter Development Mailinglist
On Mon, Dec 14, 2009 at 1:16 PM, Patrick McHardy <kaber@trash.net> wrote:
> Adam Huffman wrote:
>> On Thu, Dec 10, 2009 at 11:01 AM, Patrick McHardy <kaber@trash.net> wrote:
>>> Eric Dumazet wrote:
>>>> Le 09/12/2009 16:11, Avi Kivity a écrit :
>>>>> On 12/09/2009 03:46 PM, Adam Huffman wrote:
>>>>>> I've been seeing lots of crashes on a new Dell Precision T7500,
>>>>>> running the KVM in Fedora 12. Finally managed to capture an Oops,
>>>>>> which is shown below (hand-transcribed):
>>>>>>
>>>>>> BUG: unable to handle kernel paging request at 0000000000200200
>>>>>> IP: [<ffffffff8139aab7>] destroy_conntrack+0x82/0x11f
>>>>>> PGD 332d0e067 PUD 33453c067 PMD 0
>>>>>> RIP: 0010:[<ffffffff8139aab7>] [<ffffffff8139aab7>]
>>>>>> destroy_conntrack+0x82/0x11f
>>>>>> RSP: 0018:ffffc90000803bf0 EFLAGS: 00010202
>>>>>> RAX: 0000000080000001 RBX: ffffffff816fb1a0 RCX: 000000000000752f
>>>>>> RDX: 0000000000200200 RSI: 0000000000000011 RDI: ffffffff816fb1a0
>>>>>> RBP: ffffc90000803c00 R08: ffff880336699438 R09: 0000000000aaa5e0
>>>>>> R10: 00000002f54189d5 R11: 0000000000000001 R12: ffffffff819a92e0
>>>>>> R13: ffffffffa029adcc R14: 0000000000000000 R15: ffff880632866c38
>>>>>> FS: 00007fdd34b17710(0000) GS:ffffc90000800000(0000)
>>>>>> knlGS:0000000000000000
>>>>>> CS: 0010 DS: 002B ES: 002B CR0: 0000000080050033
>>>>>> CR2: 0000000000200200 CR3: 00000003349c0000 CR4: 00000000000026e0
>>>>>> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>>>>> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>>>>>> Process qemu-kvm (pid: 1759, threadinfo ffff88062e9e8000, task
>>>>>> ffff880634945e00)
>>>>>> Stack:
>>>>>> ffff880632866c00 ffff880634640c30 ffffc90000803c10 ffffffff813989c2
>>>>>> <0> ffffc90000803c30 ffffffff81374092 ffffc90000803c30 ffff880632866c00
>>>>>> <0> ffffc90000803c50 ffffffff81373dd3 0000000200000000 ffff880632866c00
>>>>>> Call Trace:
>>>>>> <IRQ>
>>>>>> [<ffffffff813989c2>] nf_conntrack_destroy+0x1b/0x1d
>>>>>> [<ffffffff81374092>] skb_release_head_state+0x95/0xd7
>>>>>> [<ffffffff81373dd3>] __kfree_skb+0x16/0x81
>>>>>> [<ffffffff81373ed7>] kfree_skb+0x6a/0x72
>>>>>> [<ffffffffa029adcc>] ip6_mc_input+0x220/0x230 [ipv6]
>>>>>> [<ffffffffa029a3d1>] ip6_rcv_finish+0x27/0x2b [ipv6]
>>>>>> [<ffffffffa029a763>] ipv6_rcv+0x38e/0x3e5 [ipv6]
>>>>>> [<ffffffff8137bd91>] netif_receive_skb+0x402/0x427
>>>>>> ...
>>>>>>
>>>> crash in :
>>>> 48 8b 43 08 mov 0x8(%rbx),%rax
>>>> a8 01 test $0x1,%al
>>>> 48 89 02 mov %rax,(%rdx) << HERE >> RDX=0x200200 (LIST_POISON2)
>>>> 75 04 jne 1f
>>>> 48 89 50 08 mov %rdx,0x8(%rax)
>>>> 1: 48 c7 43 10 00 02 20 movq $0x200200,0x10(%rbx)
>>>>
>>>> if (!nf_ct_is_confirmed(ct)) {
>>>> BUG_ON(hlist_nulls_unhashed(&ct->tuplehash[IP_CT_DIR_ORIGINAL].hnnode));
>>>> hlist_nulls_del_rcu(&ct->tuplehash[IP_CT_DIR_ORIGINAL].hnnode); << HERE >>
>>>> }
>>>> NF_CT_STAT_INC(net, delete);
>>>
>>> I can't spot the problem. Adam, please send me your .config file.
>>>
>>>
>>
>> It's the standard Fedora .config, which is attached.
>>
>> As I stated in another message, the oops seems related to VT-d. With
>> that disabled, the machine has been stable for nearly a day now.
>
> That probably only affects the timing of some race. Please also
> send me the IPv6 ruleset used on that machine. Thanks.
>
Again, it's the Fedora 12 default. Here you go:
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p ipv6-icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp6-adm-prohibited
-A FORWARD -j REJECT --reject-with icmp6-adm-prohibited
COMMIT
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Networking-related crash?
2009-12-14 13:16 ` Patrick McHardy
2009-12-14 13:46 ` Adam Huffman
@ 2010-01-05 16:57 ` Adam Huffman
1 sibling, 0 replies; 8+ messages in thread
From: Adam Huffman @ 2010-01-05 16:57 UTC (permalink / raw)
To: Patrick McHardy
Cc: Eric Dumazet, Avi Kivity, kvm, netdev,
Netfilter Development Mailinglist
On Mon, Dec 14, 2009 at 1:16 PM, Patrick McHardy <kaber@trash.net> wrote:
> Adam Huffman wrote:
>> On Thu, Dec 10, 2009 at 11:01 AM, Patrick McHardy <kaber@trash.net> wrote:
>>> Eric Dumazet wrote:
>>>> Le 09/12/2009 16:11, Avi Kivity a écrit :
>>>>> On 12/09/2009 03:46 PM, Adam Huffman wrote:
>>>>>> I've been seeing lots of crashes on a new Dell Precision T7500,
>>>>>> running the KVM in Fedora 12. Finally managed to capture an Oops,
>>>>>> which is shown below (hand-transcribed):
>>>>>>
>>>>>> BUG: unable to handle kernel paging request at 0000000000200200
>>>>>> IP: [<ffffffff8139aab7>] destroy_conntrack+0x82/0x11f
>>>>>> PGD 332d0e067 PUD 33453c067 PMD 0
>>>>>> RIP: 0010:[<ffffffff8139aab7>] [<ffffffff8139aab7>]
>>>>>> destroy_conntrack+0x82/0x11f
>>>>>> RSP: 0018:ffffc90000803bf0 EFLAGS: 00010202
>>>>>> RAX: 0000000080000001 RBX: ffffffff816fb1a0 RCX: 000000000000752f
>>>>>> RDX: 0000000000200200 RSI: 0000000000000011 RDI: ffffffff816fb1a0
>>>>>> RBP: ffffc90000803c00 R08: ffff880336699438 R09: 0000000000aaa5e0
>>>>>> R10: 00000002f54189d5 R11: 0000000000000001 R12: ffffffff819a92e0
>>>>>> R13: ffffffffa029adcc R14: 0000000000000000 R15: ffff880632866c38
>>>>>> FS: 00007fdd34b17710(0000) GS:ffffc90000800000(0000)
>>>>>> knlGS:0000000000000000
>>>>>> CS: 0010 DS: 002B ES: 002B CR0: 0000000080050033
>>>>>> CR2: 0000000000200200 CR3: 00000003349c0000 CR4: 00000000000026e0
>>>>>> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>>>>> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>>>>>> Process qemu-kvm (pid: 1759, threadinfo ffff88062e9e8000, task
>>>>>> ffff880634945e00)
>>>>>> Stack:
>>>>>> ffff880632866c00 ffff880634640c30 ffffc90000803c10 ffffffff813989c2
>>>>>> <0> ffffc90000803c30 ffffffff81374092 ffffc90000803c30 ffff880632866c00
>>>>>> <0> ffffc90000803c50 ffffffff81373dd3 0000000200000000 ffff880632866c00
>>>>>> Call Trace:
>>>>>> <IRQ>
>>>>>> [<ffffffff813989c2>] nf_conntrack_destroy+0x1b/0x1d
>>>>>> [<ffffffff81374092>] skb_release_head_state+0x95/0xd7
>>>>>> [<ffffffff81373dd3>] __kfree_skb+0x16/0x81
>>>>>> [<ffffffff81373ed7>] kfree_skb+0x6a/0x72
>>>>>> [<ffffffffa029adcc>] ip6_mc_input+0x220/0x230 [ipv6]
>>>>>> [<ffffffffa029a3d1>] ip6_rcv_finish+0x27/0x2b [ipv6]
>>>>>> [<ffffffffa029a763>] ipv6_rcv+0x38e/0x3e5 [ipv6]
>>>>>> [<ffffffff8137bd91>] netif_receive_skb+0x402/0x427
>>>>>> ...
>>>>>>
>>>> crash in :
>>>> 48 8b 43 08 mov 0x8(%rbx),%rax
>>>> a8 01 test $0x1,%al
>>>> 48 89 02 mov %rax,(%rdx) << HERE >> RDX=0x200200 (LIST_POISON2)
>>>> 75 04 jne 1f
>>>> 48 89 50 08 mov %rdx,0x8(%rax)
>>>> 1: 48 c7 43 10 00 02 20 movq $0x200200,0x10(%rbx)
>>>>
>>>> if (!nf_ct_is_confirmed(ct)) {
>>>> BUG_ON(hlist_nulls_unhashed(&ct->tuplehash[IP_CT_DIR_ORIGINAL].hnnode));
>>>> hlist_nulls_del_rcu(&ct->tuplehash[IP_CT_DIR_ORIGINAL].hnnode); << HERE >>
>>>> }
>>>> NF_CT_STAT_INC(net, delete);
>>>
>>> I can't spot the problem. Adam, please send me your .config file.
>>>
>>>
>>
>> It's the standard Fedora .config, which is attached.
>>
>> As I stated in another message, the oops seems related to VT-d. With
>> that disabled, the machine has been stable for nearly a day now.
>
> That probably only affects the timing of some race. Please also
> send me the IPv6 ruleset used on that machine. Thanks.
>
Just to note that if I disable IPv6 completely, the machine is stable
- certainly compared with the crashes after a few minutes when IPv6 is
enabled.
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2010-01-05 16:57 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <608c44bf0912090546s446bf973ne408e99661fdc56f@mail.gmail.com>
2009-12-09 15:11 ` Networking-related crash? Avi Kivity
2009-12-09 20:36 ` Adam Huffman
2009-12-09 20:54 ` Eric Dumazet
2009-12-10 11:01 ` Patrick McHardy
2009-12-10 15:18 ` Adam Huffman
2009-12-14 13:16 ` Patrick McHardy
2009-12-14 13:46 ` Adam Huffman
2010-01-05 16:57 ` Adam Huffman
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).