linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Dave Jones <davej@redhat.com>
To: Chris Mason <clm@fb.com>
Cc: "Linus Torvalds" <torvalds@linux-foundation.org>,
	"Mike Galbraith" <umgwanakikbuti@gmail.com>,
	"Ingo Molnar" <mingo@kernel.org>,
	"Peter Zijlstra" <peterz@infradead.org>,
	"Dâniel Fraga" <fragabr@gmail.com>,
	"Sasha Levin" <sasha.levin@oracle.com>,
	"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
	"Linux Kernel Mailing List" <linux-kernel@vger.kernel.org>
Subject: Re: frequent lockups in 3.18rc4
Date: Wed, 3 Dec 2014 13:41:11 -0500	[thread overview]
Message-ID: <20141203184111.GA32005@redhat.com> (raw)
In-Reply-To: <1417540493.21136.3@mail.thefacebook.com>

On Tue, Dec 02, 2014 at 12:14:53PM -0500, Chris Mason wrote:
 > On Tue, Dec 2, 2014 at 11:33 AM, Linus Torvalds 
 > <torvalds@linux-foundation.org> wrote:
 > > On Tue, Dec 2, 2014 at 6:13 AM, Mike Galbraith 
 > > <umgwanakikbuti@gmail.com> wrote:
 > > 
 > > At the same time, the whole "incapacitated by the rt throttle long
 > > enough for the hard lockup detector to trigger" commentary about that
 > > skip_clock_update issue does make me go "Hmmm..". It would certainly
 > > explain Dave's incomprehensible watchdog messages..
 > 
 > Dave's first email mentioned that he had panic on softlockup enabled, 
 > but even with that off the box wasn't recovering.

Not sure if I mentioned in an earlier post, but when I'm local to the machine,
I've disabled reboot-on-lockup, but yes the problem case is the
situation where it actually does lock up afterwards.

 > In my trinity runs here, I've gotten softlockup warnings where the box 
 > eventually recovered.  I'm wondering if some of the "bad" commits in 
 > the bisection are really false positives where the box would have been 
 > able to recover if we'd killed off all the trinity procs and given it 
 > time to breath.

So I've done multiple runs against 3.17-rc1 during bisecting, and hit
the case you describe, where I get a dump like below, and then it
eventually recovers. (Trinity then exits because the taint flag
changes).

I've been stuck on this kernel for a few days now trying to prove it
good/bad one way or the other, and I'm leaning towards good, given
that it recovers, even though the traces look similar.

	Dave


[ 9862.915562] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [trinity-c29:13237]
[ 9862.915684] Modules linked in: 8021q garp stp tun fuse bnep hidp llc2 af_key nfnetlink can_bcm scsi_transport_iscsi can_raw nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc rfcomm bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep e1000e snd_seq coretemp hwmon x86_pkg_temp_thermal nfsd kvm_intel kvm snd_seq_device snd_pcm snd_timer ptp auth_rpcgss snd shpchp oid_registry crct10dif_pclmul crc32c_intel ghash_clmulni_intel usb_debug soundcore pps_core nfs_acl microcode serio_raw pcspkr lockd sunrpc
[ 9862.915987] CPU: 0 PID: 13237 Comm: trinity-c29 Not tainted 3.17.0-rc1+ #112
[ 9862.916046] task: ffff88022657dbc0 ti: ffff8800962b0000 task.ti: ffff8800962b0000
[ 9862.916071] RIP: 0010:[<ffffffff81042569>]  [<ffffffff81042569>] lookup_address_in_pgd+0x89/0xe0
[ 9862.916103] RSP: 0018:ffff8800962b36a8  EFLAGS: 00000202
[ 9862.917024] RAX: ffff88024da748d0 RBX: ffffffff81164c63 RCX: 0000000000000001
[ 9862.917956] RDX: ffff8800962b3740 RSI: ffff8801a3417000 RDI: 000000024da74000
[ 9862.918891] RBP: ffff8800962b36a8 R08: 00003ffffffff000 R09: ffff880000000000
[ 9862.919828] R10: 0000000000000001 R11: 0000000000000000 R12: ffffffff81375a47
[ 9862.920758] R13: ffff8800962b3618 R14: ffff8802441d81f0 R15: fffff70c86134602
[ 9862.921681] FS:  00007f06c569f740(0000) GS:ffff880244000000(0000) knlGS:0000000000000000
[ 9862.922603] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 9862.923526] CR2: 0000000002378590 CR3: 00000000a1ba3000 CR4: 00000000001407f0
[ 9862.924459] DR0: 00007f40a579e000 DR1: 0000000000000000 DR2: 0000000000000000
[ 9862.925386] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[ 9862.926342] Stack:
[ 9862.927255]  ffff8800962b36b8 ffffffff810425e8 ffff8800962b36c8 ffffffff810426ab
[ 9862.928188]  ffff8800962b37c0 ffffffff810427a0 ffffffff810bfc5e ffff8800962b36f0
[ 9862.929127]  ffffffff810a89f5 ffff8800962b3768 ffffffff810c19b4 0000000000000002
[ 9862.930072] Call Trace:
[ 9862.931008]  [<ffffffff810425e8>] lookup_address+0x28/0x30
[ 9862.931958]  [<ffffffff810426ab>] _lookup_address_cpa.isra.9+0x3b/0x40
[ 9862.932913]  [<ffffffff810427a0>] __change_page_attr_set_clr+0xf0/0xab0
[ 9862.933869]  [<ffffffff810bfc5e>] ? put_lock_stats.isra.23+0xe/0x30
[ 9862.934831]  [<ffffffff810a89f5>] ? local_clock+0x25/0x30
[ 9862.935827]  [<ffffffff810c19b4>] ? __lock_acquire.isra.31+0x264/0xa60
[ 9862.936798]  [<ffffffff8109bfed>] ? finish_task_switch+0x7d/0x120
[ 9862.937765]  [<ffffffff810bfc5e>] ? put_lock_stats.isra.23+0xe/0x30
[ 9862.938730]  [<ffffffff810bffc6>] ? lock_release_holdtime.part.24+0xe6/0x160
[ 9862.939698]  [<ffffffff8104418b>] kernel_map_pages+0x7b/0x120
[ 9862.940653]  [<ffffffff81178517>] get_page_from_freelist+0x497/0xaa0
[ 9862.941597]  [<ffffffff81179498>] __alloc_pages_nodemask+0x228/0xb20
[ 9862.942539]  [<ffffffff810a89f5>] ? local_clock+0x25/0x30
[ 9862.943469]  [<ffffffff810c19b4>] ? __lock_acquire.isra.31+0x264/0xa60
[ 9862.944411]  [<ffffffff8135fe50>] ? __radix_tree_preload+0x60/0xf0
[ 9862.945357]  [<ffffffff8135fe50>] ? __radix_tree_preload+0x60/0xf0
[ 9862.946326]  [<ffffffff811c14c1>] alloc_pages_vma+0xf1/0x1b0
[ 9862.947263]  [<ffffffff8118766e>] ? shmem_alloc_page+0x6e/0xc0
[ 9862.948205]  [<ffffffff8118766e>] shmem_alloc_page+0x6e/0xc0
[ 9862.949148]  [<ffffffff810a2dbb>] ? preempt_count_sub+0x6b/0xf0
[ 9862.950090]  [<ffffffff810a2c5d>] ? get_parent_ip+0xd/0x50
[ 9862.951012]  [<ffffffff810a2dbb>] ? preempt_count_sub+0x6b/0xf0
[ 9862.951913]  [<ffffffff81382666>] ? __percpu_counter_add+0x86/0xb0
[ 9862.952811]  [<ffffffff811a4362>] ? __vm_enough_memory+0x62/0x1c0
[ 9862.953700]  [<ffffffff812eb0c7>] ? cap_vm_enough_memory+0x47/0x50
[ 9862.954591]  [<ffffffff81189f00>[ 9893.337880] [sched_delayed] sched: RT throttling activated
[ 9918.893057] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 24s! [swapper/1:0]
[ 9918.894352] Modules linked in: 8021q garp stp tun fuse bnep hidp llc2 af_key nfnetlink can_bcm scsi_transport_iscsi can_raw nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc rfcomm bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep e1000e snd_seq coretemp hwmon x86_pkg_temp_thermal nfsd kvm_intel kvm snd_seq_device snd_pcm snd_timer ptp auth_rpcgss snd shpchp oid_registry crct10dif_pclmul crc32c_intel ghash_clmulni_intel usb_debug soundcore pps_core nfs_acl microcode serio_raw pcspkr lockd sunrpc
[ 9918.901158] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G             L 3.17.0-rc1+ #112
[ 9918.903863] task: ffff880242b716f0 ti: ffff88024240c000 task.ti: ffff88024240c000
[ 9918.905218] RIP: 0010:[<ffffffff81645849>]  [<ffffffff81645849>] cpuidle_enter_state+0x79/0x1c0
[ 9918.906591] RSP: 0000:ffff88024240fe60  EFLAGS: 00000246
[ 9918.907933] RAX: 0000000000000000 RBX: ffff880242b716f0 RCX: 0000000000000019
[ 9918.909264] RDX: 20c49ba5e353f7cf RSI: 000000000003cea4 RDI: 00536da522b45eb6
[ 9918.910594] RBP: ffff88024240fe98 R08: 000000008baf9f86 R09: 0000000000000000
[ 9918.911916] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88024240fdf0
[ 9918.913243] R13: ffffffff810bfc5e R14: ffff88024240fdd0 R15: 00000000000001e1
[ 9918.914523] FS:  0000000000000000(0000) GS:ffff880244200000(0000) knlGS:0000000000000000
[ 9918.915815] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 9918.917108] CR2: 000000000044bfa0 CR3: 0000000001c11000 CR4: 00000000001407e0
[ 9918.918414] DR0: 00007f40a579e000 DR1: 0000000000000000 DR2: 0000000000000000
[ 9918.919710] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[ 9918.920998] Stack:
[ 9918.922273]  00000906c89bd5bb ffffffff81cae7f0 ffffffff81d1d290 ffffe8ffff004da8
[ 9918.923576]  ffff88024240c000 ffffffff81cae620 ffff88024240c000 ffff88024240fea8
[ 9918.924845]  ffffffff81645a47 ffff88024240ff10 ffffffff810b9fb4 ffff88024240ffd8
[ 9918.926108] Call Trace:
[ 9918.927351]  [<ffffffff81645a47>] cpuidle_enter+0x17/0x20
[ 9918.928602]  [<ffffffff810b9fb4>] cpu_startup_entry+0x384/0x410
[ 9918.929850]  [<ffffffff8102ff37>] start_secondary+0x237/0x340
[ 9918.931086] Code: d0 48 89 df ff 50 48 41 89 c5 e8 b3 5c aa ff 44 8b 63 04 49 89 c7 0f 1f 44 00 00 e8 a2 19 b0 ff fb 48 ba cf f7 53 e3 a5 9b c4 20 <4c> 2b 7d c8 4c 89 f8 49 c1 ff 3f 48 f7 ea b8 ff ff ff 7f 48 c1 
[ 9918.933755] sending NMI to other CPUs:
[ 9918.935008] NMI backtrace for cpu 2
[ 9918.936208] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G             L 3.17.0-rc1+ #112
[ 9918.938592] task: ffff880242b744d0 ti: ffff880242414000 task.ti: ffff880242414000
[ 9918.939793] RIP: 0010:[<ffffffff813c9e65>]  [<ffffffff813c9e65>] intel_idle+0xd5/0x180
[ 9918.941003] RSP: 0018:ffff880242417e20  EFLAGS: 00000046
[ 9918.942191] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[ 9918.943369] RDX: 0000000000000000 RSI: ffff880242417fd8 RDI: 0000000000000002
[ 9918.944535] RBP: ffff880242417e50 R08: 000000008baf9f86 R09: 0000000000000000
[ 9918.945695] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[ 9918.946856] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880242414000
[ 9918.948013] FS:  0000000000000000(0000) GS:ffff880244400000(0000) knlGS:0000000000000000
[ 9918.949175] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 9918.950339] CR2: 000000000068f760 CR3: 0000000001c11000 CR4: 00000000001407e0
[ 9918.951515] DR0: 00007f40a579e000 DR1: 0000000000000000 DR2: 0000000000000000
[ 9918.952678] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[ 9918.953840] Stack:
[ 9918.954988]  0000000242414000 d7251be6f43cb9e7 ffffe8ffff204da8 0000000000000005
[ 9918.956164]  ffffffff81cae620 0000000000000002 ffff880242417e98 ffffffff81645825
[ 9918.957345]  00000906cb95d0a1 ffffffff81cae7f0 ffffffff81d1d290 ffffe8ffff204da8
[ 9918.958527] Call Trace:
[ 9918.959692]  [<ffffffff81645825>] cpuidle_enter_state+0x55/0x1c0
[ 9918.960866]  [<ffffffff81645a47>] cpuidle_enter+0x17/0x20
[ 9918.962029]  [<ffffffff810b9fb4>] cpu_startup_entry+0x384/0x410
[ 9918.963185]  [<ffffffff8102ff37>] start_secondary+0x237/0x340
[ 9918.964339] Code: 31 d2 65 48 8b 34 25 08 ba 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 08 ba 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 
[ 9918.966852] NMI backtrace for cpu 3
[ 9918.968059] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G             L 3.17.0-rc1+ #112
[ 9918.970450] task: ffff880242b72de0 ti: ffff880242418000 task.ti: ffff880242418000
[ 9918.971652] RIP: 0010:[<ffffffff813c9e65>]  [<ffffffff813c9e65>] intel_idle+0xd5/0x180
[ 9918.972862] RSP: 0018:ffff88024241be20  EFLAGS: 00000046
[ 9918.974045] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001
[ 9918.975212] RDX: 0000000000000000 RSI: ffff88024241bfd8 RDI: 0000000000000003
[ 9918.976349] RBP: ffff88024241be50 R08: 000000008baf9f86 R09: 0000000000000000
[ 9918.977463] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000005
[ 9918.978550] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880242418000
[ 9918.979628] FS:  0000000000000000(0000) GS:ffff880244600000(0000) knlGS:0000000000000000
[ 9918.980695] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 9918.981739] CR2: 00007f37d766f050 CR3: 0000000001c11000 CR4: 00000000001407e0
[ 9918.982786] DR0: 00007f40a579e000 DR1: 0000000000000000 DR2: 0000000000000000
[ 9918.983821] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[ 9918.984849] Stack:
[ 9918.985866]  0000000342418000 63d0570b22f343d2 ffffe8ffff404da8 0000000000000005
[ 9918.986911]  ffffffff81cae620 0000000000000003 ffff88024241be98 ffffffff81645825
[ 9918.987961]  00000906cb9636ab ffffffff81cae7f0 ffffffff81d1d290 ffffe8ffff404da8
[ 9918.989007] Call Trace:
[ 9918.990036]  [<ffffffff81645825>] cpuidle_enter_state+0x55/0x1c0
[ 9918.991072]  [<ffffffff81645a47>] cpuidle_enter+0x17/0x20
[ 9918.992097]  [<ffffffff810b9fb4>] cpu_startup_entry+0x384/0x410
[ 9918.993127]  [<ffffffff8102ff37>] start_secondary+0x237/0x340
[ 9918.994152] Code: 31 d2 65 48 8b 34 25 08 ba 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 


  reply	other threads:[~2014-12-03 18:42 UTC|newest]

Thread overview: 486+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2014-11-14 21:31 frequent lockups in 3.18rc4 Dave Jones
2014-11-14 22:01 ` Linus Torvalds
2014-11-14 22:30   ` Dave Jones
2014-11-14 22:55   ` Thomas Gleixner
2014-11-14 23:32     ` Dave Jones
2014-11-15  0:36       ` Thomas Gleixner
2014-11-15  2:40         ` Dave Jones
2014-11-16 12:16           ` Thomas Gleixner
2014-11-15  1:59     ` Linus Torvalds
2014-11-17 21:22       ` Linus Torvalds
2014-11-17 22:31         ` Thomas Gleixner
2014-11-17 22:43           ` Thomas Gleixner
2014-11-17 22:58             ` Jens Axboe
2014-11-17 23:59             ` Linus Torvalds
2014-11-18  0:15               ` Thomas Gleixner
2014-11-17 23:04         ` Jens Axboe
2014-11-17 23:17           ` Thomas Gleixner
2014-11-18  2:23             ` Jens Axboe
2014-11-15 21:34   ` Dave Jones
2014-11-16  1:40     ` Dave Jones
2014-11-16  6:33       ` Linus Torvalds
2014-11-16 10:06         ` Markus Trippelsdorf
2014-11-16 18:33           ` Linus Torvalds
2014-11-17 17:03         ` Dave Jones
2014-11-17 19:59           ` Linus Torvalds
2014-11-18  2:09             ` Dave Jones
2014-11-18  2:21               ` Linus Torvalds
2014-11-18  2:39                 ` Dave Jones
2014-11-18  2:51                   ` Linus Torvalds
2014-11-18 14:52                     ` Dave Jones
2014-11-18 17:20                       ` Linus Torvalds
2014-11-18 19:28                         ` Thomas Gleixner
2014-11-18 21:25                           ` Don Zickus
2014-11-18 21:31                             ` Dave Jones
2014-11-18 18:54                       ` Thomas Gleixner
2014-11-18 21:55                         ` Don Zickus
2014-11-18 22:02                           ` Dave Jones
2014-11-19 14:41                             ` Don Zickus
2014-11-19 15:03                               ` Vivek Goyal
2014-11-19 15:38                                 ` Dave Jones
2014-11-19 16:28                                   ` Vivek Goyal
2014-11-20 16:10                                     ` Dave Jones
2014-11-20 16:48                                       ` Vivek Goyal
2014-11-20 17:38                                         ` Dave Jones
2014-11-21  9:46                                           ` Dave Young
2014-11-20 16:54                                       ` Vivek Goyal
2014-11-20  9:54                               ` Dave Young
2014-11-19  2:19                           ` Dave Jones
2014-11-19  4:40                             ` Linus Torvalds
2014-11-19  4:59                               ` Dave Jones
2014-11-19  5:15                               ` Dave Jones
2014-11-20 14:36                                 ` Frederic Weisbecker
2014-11-19 14:59                               ` Dave Jones
2014-11-19 17:22                                 ` Linus Torvalds
2014-11-19 17:40                                   ` Linus Torvalds
2014-11-19 19:02                                     ` Frederic Weisbecker
2014-11-19 19:03                                       ` Andy Lutomirski
2014-11-19 23:00                                         ` Frederic Weisbecker
2014-11-19 23:07                                           ` Andy Lutomirski
2014-11-19 23:13                                             ` Frederic Weisbecker
2014-11-19 21:56                                       ` Thomas Gleixner
2014-11-19 22:56                                         ` Frederic Weisbecker
2014-11-19 22:59                                           ` Andy Lutomirski
2014-11-19 23:07                                             ` Frederic Weisbecker
2014-11-19 23:09                                           ` Thomas Gleixner
2014-11-19 23:50                                             ` Frederic Weisbecker
2014-11-20 12:23                                               ` Tejun Heo
2014-11-20 21:58                                                 ` Thomas Gleixner
2014-11-20 22:06                                                   ` Andy Lutomirski
2014-11-20 22:11                                                   ` Tejun Heo
2014-11-20 22:42                                                     ` Thomas Gleixner
2014-11-20 23:05                                                       ` Tejun Heo
2014-11-20 23:08                                                         ` Andy Lutomirski
2014-11-20 23:34                                                           ` Linus Torvalds
2014-11-20 23:39                                                           ` Tejun Heo
2014-11-20 23:55                                                             ` Andy Lutomirski
2014-11-21 16:27                                                               ` Tejun Heo
2014-11-21 16:38                                                                 ` Andy Lutomirski
2014-11-21 16:48                                                                   ` Linus Torvalds
2014-11-21 17:08                                                                     ` Steven Rostedt
2014-11-21 17:19                                                                       ` Linus Torvalds
2014-11-21 17:22                                                                         ` Andy Lutomirski
2014-11-21 18:22                                                                           ` Linus Torvalds
2014-11-21 18:28                                                                             ` Andy Lutomirski
2014-11-21 19:06                                                                             ` Linus Torvalds
2014-11-21 19:23                                                                               ` Steven Rostedt
2014-11-21 19:34                                                                                 ` Linus Torvalds
2014-11-21 19:46                                                                                   ` Linus Torvalds
2014-11-21 19:52                                                                                     ` Andy Lutomirski
2014-11-21 20:14                                                                                       ` Josh Boyer
2014-11-21 20:16                                                                                         ` Andy Lutomirski
2014-11-21 20:23                                                                                           ` Josh Boyer
2014-11-24 18:48                                                                                             ` Konrad Rzeszutek Wilk
2014-11-24 19:07                                                                                               ` Josh Boyer
2014-11-25  5:36                                                                                               ` Jürgen Groß
2014-11-25 17:22                                                                                                 ` Linus Torvalds
2014-11-21 20:00                                                                                     ` Dave Jones
2014-11-21 20:02                                                                                       ` Andy Lutomirski
2014-11-21 19:51                                                                               ` Thomas Gleixner
2014-11-21 20:00                                                                                 ` Linus Torvalds
2014-11-21 20:16                                                                                   ` Thomas Gleixner
2014-11-21 20:41                                                                                     ` Linus Torvalds
2014-11-21 21:11                                                                                       ` Thomas Gleixner
2014-11-21 22:55                                                                                         ` Linus Torvalds
2014-11-21 23:03                                                                                           ` Andy Lutomirski
2014-11-21 23:33                                                                                             ` Linus Torvalds
2014-12-16 19:28                                                                                           ` Peter Zijlstra
2014-12-16 20:46                                                                                             ` Linus Torvalds
2014-12-16 21:19                                                                                               ` Mel Gorman
2014-12-16 23:02                                                                                                 ` Peter Zijlstra
2014-12-17  0:00                                                                                                   ` Linus Torvalds
2014-12-17  0:41                                                                                                     ` Andy Lutomirski
2014-12-17 17:01                                                                                                       ` Konrad Rzeszutek Wilk
2014-12-17 17:14                                                                                                         ` Peter Zijlstra
2014-11-21 22:33                                                                                 ` Konrad Rzeszutek Wilk
2014-11-22  1:17                                                                                   ` Thomas Gleixner
2014-11-21 17:34                                                                         ` Steven Rostedt
2014-11-21 18:24                                                                           ` Linus Torvalds
2014-11-21 22:10                                                                   ` Frederic Weisbecker
2014-11-21  2:33                                                             ` Steven Rostedt
2014-11-21  0:54                                                         ` Thomas Gleixner
2014-11-21 14:13                                                           ` Frederic Weisbecker
2014-11-21 16:25                                                             ` Tejun Heo
2014-11-21 17:01                                                               ` Steven Rostedt
2014-11-21 17:11                                                                 ` Steven Rostedt
2014-11-21 21:32                                                                 ` Frederic Weisbecker
2014-11-21 21:34                                                                   ` Andy Lutomirski
2014-11-21 21:50                                                                     ` Frederic Weisbecker
2014-11-21 22:45                                                                       ` Steven Rostedt
2014-11-21 21:44                                                               ` Frederic Weisbecker
2014-11-22  0:11                                                                 ` Tejun Heo
2014-11-22  0:18                                                                   ` Linus Torvalds
2014-11-22  0:41                                                                     ` Andy Lutomirski
2014-11-19 23:54                                             ` Andy Lutomirski
2014-11-20  0:00                                               ` Thomas Gleixner
2014-11-20  0:30                                                 ` Andy Lutomirski
2014-11-20  0:40                                                   ` Linus Torvalds
2014-11-20  0:49                                                     ` Andy Lutomirski
2014-11-20  1:07                                                       ` Linus Torvalds
2014-11-20  1:16                                                         ` Andy Lutomirski
2014-11-20  2:42                                                           ` Linus Torvalds
2014-11-20  6:16                                                             ` Andy Lutomirski
2014-11-19 19:15                                   ` Andy Lutomirski
2014-11-19 19:38                                     ` Linus Torvalds
2014-11-19 22:18                                       ` Dave Jones
2014-11-19 21:01                                 ` Andy Lutomirski
2014-11-19 21:47                                   ` Dave Jones
2014-11-19 21:58                                     ` Borislav Petkov
2014-11-19 22:18                                       ` Dave Jones
2014-11-20 10:33                                         ` Borislav Petkov
2014-11-19 21:56                                   ` [PATCH] x86, syscall: Fix _TIF_NOHZ handling in syscall_trace_enter_phase1 Andy Lutomirski
2014-11-19 22:13                                     ` Thomas Gleixner
2014-11-20 20:33                                       ` Linus Torvalds
2014-11-20 22:07                                         ` Thomas Gleixner
2014-11-20 22:04                                     ` [tip:x86/urgent] " tip-bot for Andy Lutomirski
2014-11-20 15:25                                   ` frequent lockups in 3.18rc4 Dave Jones
2014-11-20 19:43                                     ` Linus Torvalds
2014-11-20 20:06                                       ` Dave Jones
2014-11-20 20:37                                       ` Don Zickus
2014-11-20 20:51                                         ` Linus Torvalds
2014-11-21  6:37                                       ` Ingo Molnar
2014-11-21 14:50                                         ` Dave Jones
2014-11-25 12:22                                     ` Will Deacon
2014-12-01 11:48                                       ` Will Deacon
2014-12-01 17:05                                         ` Linus Torvalds
2014-12-01 17:10                                           ` Will Deacon
2014-12-01 17:53                                             ` Linus Torvalds
2014-12-01 18:25                                               ` Kirill A. Shutemov
2014-12-01 18:36                                                 ` Linus Torvalds
2014-12-04 10:51                                                   ` Will Deacon
2014-12-04 14:56                                                     ` Dave Jones
2014-12-05 13:49                                                       ` Will Deacon
2014-11-20 15:04                                 ` Frederic Weisbecker
2014-11-20 15:08           ` Frederic Weisbecker
2014-11-20 16:19             ` Dave Jones
2014-11-20 16:42               ` Frederic Weisbecker
2014-11-26  0:25         ` Dave Jones
2014-11-26  1:48           ` Linus Torvalds
2014-11-26  2:40             ` Dave Jones
2014-11-26 22:57               ` Dave Jones
2014-11-27  0:46                 ` Linus Torvalds
2014-11-27 19:17                 ` Linus Torvalds
2014-11-27 22:56                   ` Dave Jones
2014-11-29 20:38                     ` Dâniel Fraga
2014-11-30 20:45                       ` Linus Torvalds
2014-11-30 21:21                         ` Dâniel Fraga
2014-12-01  0:21                           ` Linus Torvalds
2014-12-01  1:02                             ` Dâniel Fraga
2014-12-01 19:14                               ` Paul E. McKenney
2014-12-01 20:28                                 ` Dâniel Fraga
2014-12-01 20:36                                   ` Linus Torvalds
2014-12-01 23:08                                     ` Chris Mason
2014-12-01 23:25                                       ` Linus Torvalds
2014-12-01 23:44                                         ` Chris Mason
2014-12-02  0:39                                           ` Linus Torvalds
2014-12-02 14:13                                       ` Mike Galbraith
2014-12-02 16:33                                         ` Linus Torvalds
2014-12-02 17:14                                           ` Chris Mason
2014-12-03 18:41                                             ` Dave Jones [this message]
2014-12-03 18:45                                               ` Linus Torvalds
2014-12-03 19:00                                                 ` Dave Jones
2014-12-03 19:25                                                   ` Linus Torvalds
2014-12-03 19:30                                                     ` Dave Jones
2014-12-03 19:48                                                     ` Linus Torvalds
2014-12-03 20:09                                                       ` Dave Jones
2014-12-03 20:37                                                         ` Linus Torvalds
2014-12-03 20:55                                                           ` Thomas Gleixner
2014-12-03 21:14                                                             ` Linus Torvalds
2014-12-03 22:19                                                               ` Thomas Gleixner
2014-12-03 23:21                                                                 ` Dave Jones
2014-12-03 23:49                                                                   ` Thomas Gleixner
2014-12-04  0:19                                                                     ` Linus Torvalds
2014-12-04  1:02                                                                       ` Thomas Gleixner
2014-12-04  0:20                                                                     ` Dave Jones
2014-12-04  0:59                                                                       ` Thomas Gleixner
2014-12-04  1:32                                                                         ` Dave Jones
2014-12-04  3:45                                                                           ` Dave Jones
2014-12-03 19:56                                                     ` John Stultz
2014-12-03 20:37                                                       ` Thomas Gleixner
2014-12-03 20:44                                                         ` Dave Jones
2014-12-03 20:59                                                           ` Thomas Gleixner
2014-12-03 21:05                                                             ` Dave Jones
2014-12-03 21:48                                                               ` Thomas Gleixner
2014-12-03 20:39                                                       ` Thomas Gleixner
2014-12-04  3:15                                                         ` Chris Mason
2014-12-04  5:49                                                           ` Linus Torvalds
2014-12-04 14:57                                                             ` Chris Mason
2014-12-04 15:22                                                             ` Dave Hansen
2014-12-04 15:30                                                               ` Chris Mason
2014-12-03 19:59                                                   ` Chris Mason
2014-12-03 20:11                                                     ` Dave Jones
2014-12-03 20:56                                                       ` Chris Mason
2014-12-04  0:27                                                 ` Dave Jones
2014-12-05 17:15                                                 ` Dave Jones
2014-12-05 18:38                                                   ` Linus Torvalds
2014-12-05 18:48                                                     ` Dave Jones
2014-12-05 19:31                                                       ` Linus Torvalds
2014-12-05 19:37                                                         ` Dave Jones
2014-12-06 22:38                                                         ` Thomas Gleixner
2014-12-06  9:37                                                       ` Chuck Ebbert
2014-12-06 16:22                                                         ` Martin van Es
2014-12-06 20:09                                                           ` Linus Torvalds
2014-12-06 20:41                                                             ` Linus Torvalds
2014-12-06 21:14                                                             ` Martin van Es
2014-12-12 12:58                                                             ` Martin van Es
2014-12-15 12:07                                                               ` Martin van Es
2014-12-06 22:14                                                         ` Thomas Gleixner
2014-12-05 19:04                                                     ` Chris Mason
2014-12-05 19:29                                                       ` Linus Torvalds
2014-12-11 14:54                                                         ` Dave Jones
2014-12-11 21:49                                                           ` Linus Torvalds
2014-12-11 21:52                                                             ` Sasha Levin
2014-12-11 21:57                                                               ` Chris Mason
2014-12-11 22:00                                                                 ` Sasha Levin
2014-12-11 22:36                                                               ` Linus Torvalds
2014-12-11 22:57                                                                 ` Sasha Levin
2014-12-12  6:54                                                                   ` Ingo Molnar
2014-12-12 23:54                                                                   ` Sasha Levin
2014-12-13  0:23                                                                     ` Linus Torvalds
2014-12-13  0:34                                                                       ` Sasha Levin
2014-12-13  0:44                                                                         ` Linus Torvalds
2014-12-13 16:28                                                                           ` Jeff Chua
2014-12-13  2:32                                                                       ` Dave Jones
2014-12-11 21:57                                                             ` Borislav Petkov
2014-12-12  3:03                                                             ` Dave Jones
2014-12-12  4:45                                                               ` Dave Jones
2014-12-12 14:38                                                                 ` Dave Jones
2014-12-12 18:24                                                                   ` Paul E. McKenney
2014-12-12 18:10                                                                 ` Paul E. McKenney
2014-12-12 18:42                                                                   ` Dave Jones
2014-12-12 18:54                                                             ` Dave Jones
2014-12-12 19:14                                                               ` Linus Torvalds
2014-12-12 19:23                                                                 ` Dave Jones
2014-12-12 19:58                                                                 ` David Lang
2014-12-12 20:20                                                                   ` Linus Torvalds
2014-12-13  7:43                                                                     ` Ingo Molnar
2014-12-12 20:34                                                                   ` Paul E. McKenney
2014-12-12 21:23                                                                     ` Sasha Levin
2014-12-13  0:58                                                                       ` Paul E. McKenney
2014-12-13 12:08                                                                         ` Paul E. McKenney
2014-12-13  8:30                                                                       ` Ingo Molnar
2014-12-13 15:53                                                                         ` Sasha Levin
2014-12-13 18:07                                                                           ` Paul E. McKenney
2014-12-14 17:50                                                                             ` Paul E. McKenney
2014-12-14 23:46                                                                               ` Sasha Levin
2014-12-15  0:11                                                                                 ` Paul E. McKenney
2014-12-15  1:20                                                                                   ` Sasha Levin
2014-12-15  6:33                                                                                     ` Paul E. McKenney
2014-12-15 12:56                                                                                       ` Paul E. McKenney
2014-12-15 13:16                                                                                         ` Sasha Levin
2014-12-16  3:40                                                                                           ` Paul E. McKenney
2014-12-13  7:36                                                                 ` [PATCH] sched: Fix lost reschedule in __cond_resched() Ingo Molnar
2014-12-14 18:04                                                                   ` Frederic Weisbecker
2014-12-14 19:43                                                                     ` Ingo Molnar
2014-12-14 19:50                                                                     ` Linus Torvalds
2014-12-14 20:30                                                                       ` Frederic Weisbecker
2014-12-13  8:19                                                                 ` frequent lockups in 3.18rc4 Ingo Molnar
2014-12-13  8:27                                                                   ` Ingo Molnar
2014-12-13 14:15                                                                     ` Sasha Levin
2014-12-13 16:59                                                                 ` Dave Jones
2014-12-13 18:04                                                                   ` Paul E. McKenney
2014-12-13 20:41                                                                     ` Dave Jones
2014-12-14  4:04                                                                       ` Paul E. McKenney
2014-12-13 22:36                                                                   ` Dave Jones
2014-12-13 22:40                                                                     ` Linus Torvalds
2014-12-13 22:59                                                                       ` Linus Torvalds
2014-12-13 23:09                                                                         ` Linus Torvalds
2014-12-13 23:35                                                                           ` Al Viro
2014-12-13 23:38                                                                             ` Linus Torvalds
2014-12-13 23:47                                                                               ` Al Viro
2014-12-14  0:14                                                                                 ` Linus Torvalds
2014-12-14  0:33                                                                                   ` Al Viro
2014-12-14  1:35                                                                                     ` Linus Torvalds
2014-12-14  3:14                                                                                       ` Al Viro
2014-12-15  0:18                                                                                         ` Al Viro
2014-12-13 23:39                                                                         ` Al Viro
2014-12-14 23:46                                                                       ` Dave Jones
2014-12-15  0:38                                                                         ` Linus Torvalds
2014-12-15  0:42                                                                           ` Dave Jones
2014-12-15  5:47                                                                           ` Linus Torvalds
2014-12-15  5:57                                                                             ` Dave Jones
2014-12-15 18:21                                                                               ` Linus Torvalds
2014-12-15 23:46                                                                                 ` Linus Torvalds
2014-12-18  2:42                                                                                   ` Sasha Levin
2014-12-18  2:45                                                                                     ` Linus Torvalds
2014-12-18  5:13                                                                                   ` Dave Jones
2014-12-18 15:54                                                                                     ` Chris Mason
2014-12-18 16:12                                                                                       ` Dave Jones
2014-12-19  2:45                                                                                         ` Dave Jones
2014-12-19  3:49                                                                                           ` Linus Torvalds
2014-12-19  3:58                                                                                             ` Dave Jones
2014-12-19  4:03                                                                                               ` Dave Jones
2014-12-19  4:48                                                                                                 ` Linus Torvalds
2014-12-19 11:35                                                                                                   ` Peter Zijlstra
2014-12-19 14:55                                                                                                   ` Dave Jones
2014-12-19 15:14                                                                                                     ` Chris Mason
2014-12-19 19:15                                                                                                     ` Linus Torvalds
2014-12-19 19:44                                                                                                       ` Peter Zijlstra
2014-12-19 19:51                                                                                                       ` Linus Torvalds
2014-12-19 20:46                                                                                                         ` Linus Torvalds
2014-12-19 20:54                                                                                                           ` Dave Jones
2014-12-19 22:05                                                                                                             ` Linus Torvalds
2014-12-20 16:49                                                                                                               ` Dave Jones
2014-12-19 20:31                                                                                                       ` Chris Mason
2014-12-19 20:36                                                                                                         ` Dave Jones
2014-12-19 23:22                                                                                                         ` Thomas Gleixner
2014-12-20  0:12                                                                                                           ` Chris Mason
2014-12-20  1:06                                                                                                             ` Thomas Gleixner
2014-12-19 23:14                                                                                                       ` Thomas Gleixner
2014-12-19 23:55                                                                                                         ` Linus Torvalds
2014-12-20  1:00                                                                                                           ` Thomas Gleixner
2014-12-20  1:57                                                                                                             ` Linus Torvalds
2014-12-20 18:25                                                                                                               ` Linus Torvalds
2014-12-20 21:16                                                                                                                 ` Linus Torvalds
2014-12-21  3:52                                                                                                                   ` Paul E. McKenney
2014-12-21 21:22                                                                                                                   ` Linus Torvalds
2014-12-21 22:19                                                                                                                     ` Linus Torvalds
2014-12-21 22:32                                                                                                                       ` Dave Jones
2014-12-21 23:58                                                                                                                         ` Linus Torvalds
2014-12-22  0:41                                                                                                                           ` Linus Torvalds
2014-12-22  0:52                                                                                                                             ` Linus Torvalds
2014-12-22  1:22                                                                                                                               ` Dave Jones
2014-12-22  3:11                                                                                                                               ` Paul E. McKenney
2014-12-22 19:47                                                                                                                             ` Linus Torvalds
2014-12-22 20:06                                                                                                                               ` Linus Torvalds
2014-12-22 22:57                                                                                                                               ` Dave Jones
2014-12-22 23:59                                                                                                                                 ` Linus Torvalds
2014-12-23 14:56                                                                                                                                   ` Dave Jones
2014-12-24 13:58                                                                                                                                     ` Sasha Levin
2014-12-24  3:01                                                                                                                                   ` Dave Jones
2014-12-26 16:34                                                                                                                                     ` Dave Jones
2014-12-26 18:12                                                                                                                                       ` Dave Jones
2014-12-26 20:57                                                                                                                                         ` Linus Torvalds
2014-12-26 21:20                                                                                                                                           ` Dave Jones
2014-12-26 22:57                                                                                                                                           ` Dave Jones
2014-12-26 23:16                                                                                                                                             ` Linus Torvalds
2014-12-27  0:36                                                                                                                                               ` Dave Jones
2014-12-27  3:14                                                                                                                                                 ` Linus Torvalds
2014-12-27 16:48                                                                                                                                                   ` Dave Jones
2014-12-26 23:30                                                                                                                                             ` Linus Torvalds
2014-12-27  0:39                                                                                                                                               ` Dave Jones
2014-12-27  2:53                                                                                                                                               ` Dave Jones
2015-01-03  0:27                                                                                                                                           ` John Stultz
2015-01-03 14:58                                                                                                                                             ` Sasha Levin
2015-01-04 19:46                                                                                                                                             ` Linus Torvalds
2015-01-06  1:17                                                                                                                                               ` John Stultz
2015-01-06  1:25                                                                                                                                                 ` Linus Torvalds
2015-01-06  2:05                                                                                                                                                   ` John Stultz
2014-12-22 23:59                                                                                                                               ` John Stultz
2014-12-23  0:46                                                                                                                                 ` Linus Torvalds
2014-12-27 20:33                                                                                                                                   ` Paul E. McKenney
2015-01-12 10:05                                                                                                                           ` Thomas Gleixner
2014-12-19 14:30                                                                                               ` Chris Mason
2014-12-19 15:12                                                                                                 ` Dave Jones
2014-12-18 18:54                                                                                       ` Linus Torvalds
2014-12-15 14:00                                                                             ` Borislav Petkov
2014-12-18 21:17                                                                             ` save_xstate_sig (Re: frequent lockups in 3.18rc4) Andy Lutomirski
2014-12-18 21:34                                                                               ` Linus Torvalds
2014-12-18 21:41                                                                                 ` Andy Lutomirski
2014-12-18 21:37                                                                               ` Dave Jones
2014-12-17 18:22                                                                           ` frequent lockups in 3.18rc4 Dave Jones
2014-12-17 18:57                                                                             ` Dave Jones
2014-12-17 19:24                                                                               ` Dave Jones
2014-12-17 19:51                                                                               ` Linus Torvalds
2014-12-17 20:16                                                                                 ` Dave Jones
2014-12-17 19:41                                                                             ` Linus Torvalds
2014-12-06  5:04                                                     ` Gene Heskett
2014-12-02 17:47                                           ` Mike Galbraith
2014-12-13  8:11                                             ` Ingo Molnar
2014-12-13  9:57                                               ` Mike Galbraith
2014-12-17 11:13                                           ` Peter Zijlstra
2014-12-02 19:32                                       ` Dave Jones
2014-12-02 23:32                                         ` Sasha Levin
2014-12-03  0:09                                           ` Linus Torvalds
2014-12-03  0:25                                             ` Sasha Levin
2014-12-05  5:00                                           ` Sasha Levin
2014-12-05  6:38                                             ` Linus Torvalds
2014-12-05 15:03                                               ` Sasha Levin
2014-12-05 18:15                                                 ` Linus Torvalds
2014-12-07 14:58                                                   ` Sasha Levin
2014-12-07 18:24                                                     ` Paul E. McKenney
2014-12-07 19:43                                                       ` Paul E. McKenney
2014-12-07 23:28                                                         ` Sasha Levin
2014-12-08  5:20                                                           ` Paul E. McKenney
2014-12-08 14:33                                                             ` Sasha Levin
2014-12-08 15:28                                                               ` Sasha Levin
2014-12-08 15:57                                                                 ` Paul E. McKenney
2014-12-08 16:34                                                                   ` Sasha Levin
2014-12-08 15:56                                                               ` Paul E. McKenney
2014-12-07 23:53                                                     ` Linus Torvalds
2014-12-02 19:31                                     ` Dave Jones
2014-12-02 21:17                                       ` Linus Torvalds
2014-12-02 20:30                                     ` Dave Jones
2014-12-02 20:48                                       ` Paul E. McKenney
2014-12-01 23:08                                   ` Paul E. McKenney
2014-12-02 16:43                                     ` Dâniel Fraga
2014-12-02 17:04                                       ` Paul E. McKenney
2014-12-02 17:14                                         ` Dâniel Fraga
2014-12-02 18:42                                           ` Paul E. McKenney
2014-12-02 18:47                                             ` Dâniel Fraga
2014-12-02 19:11                                               ` Paul E. McKenney
2014-12-02 19:24                                                 ` Dâniel Fraga
2014-12-02 20:56                                                   ` Paul E. McKenney
2014-12-02 22:01                                                     ` Dâniel Fraga
2014-12-02 22:10                                                       ` Paul E. McKenney
2014-12-02 22:18                                                         ` Dâniel Fraga
2014-12-02 22:35                                                           ` Paul E. McKenney
2014-12-02 22:10                                                       ` Linus Torvalds
2014-12-02 22:16                                                         ` Dâniel Fraga
2014-12-03  3:21                                                         ` Dâniel Fraga
2014-12-03  4:14                                                           ` Linus Torvalds
2014-12-03  4:51                                                             ` Dâniel Fraga
2014-12-03  6:02                                                             ` Chris Rorvick
2014-12-03 15:22                                                               ` Linus Torvalds
2014-12-04  8:43                                                                 ` Dâniel Fraga
2014-12-04 16:18                                                                   ` Linus Torvalds
2014-12-04 16:52                                                                     ` Frederic Weisbecker
2014-12-04 17:25                                                                       ` Dâniel Fraga
2014-12-04 17:47                                                                         ` Linus Torvalds
2014-12-04 18:07                                                                           ` Dâniel Fraga
2014-12-03 14:54                                                             ` Tejun Heo
2014-12-02 18:09                                         ` Paul E. McKenney
2014-12-02 18:41                                           ` Dâniel Fraga
2014-12-02 17:08                                       ` Linus Torvalds
2014-12-02 17:16                                         ` Dâniel Fraga
2014-12-02  8:40                                 ` Lai Jiangshan
2014-12-02 16:58                                   ` Paul E. McKenney
2014-12-02 16:58                                   ` Dâniel Fraga
2014-12-02 17:17                                     ` Paul E. McKenney
2014-12-03  2:03                                     ` Lai Jiangshan
2014-12-03  5:22                                       ` Paul E. McKenney
2014-12-01 16:56                     ` Don Zickus
2014-11-26  4:39             ` Jürgen Groß
     [not found]               ` <CA+55aFx1SiFBzmA=k9jHxi3cZE3Ei_+2NHepujgf86KEvkz8eQ@mail.gmail.com>
2014-11-26  5:11                 ` Dave Jones
2014-11-26  5:24                 ` Juergen Gross
2014-11-26  5:52                   ` Linus Torvalds
2014-11-26  6:21                     ` Linus Torvalds
2014-11-26  6:52                       ` Juergen Gross
2014-11-26  9:44                       ` Juergen Gross
2014-11-26 14:34                       ` Dave Jones
2014-11-26 17:37                         ` Linus Torvalds
2014-11-20 15:28       ` Frederic Weisbecker
2014-11-17 15:07 ` Don Zickus
  -- strict thread matches above, loose matches on Subject: below --
2014-12-16  3:04 Hillf Danton
2015-02-12 11:09 Martin van Es
2015-02-12 16:01 ` Linus Torvalds

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20141203184111.GA32005@redhat.com \
    --to=davej@redhat.com \
    --cc=clm@fb.com \
    --cc=fragabr@gmail.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@kernel.org \
    --cc=paulmck@linux.vnet.ibm.com \
    --cc=peterz@infradead.org \
    --cc=sasha.levin@oracle.com \
    --cc=torvalds@linux-foundation.org \
    --cc=umgwanakikbuti@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).