From: Avi Kivity <avi@redhat.com>
To: ben@iagu.net
Cc: Eric Dumazet <eric.dumazet@gmail.com>,
Thomas Gleixner <tglx@linutronix.de>,
KVM list <kvm@vger.kernel.org>,
linux-kernel <linux-kernel@vger.kernel.org>,
John Stultz <johnstul@us.ibm.com>,
Richard Cochran <richard.cochran@omicron.at>,
Peter Zijlstra <a.p.zijlstra@chello.nl>
Subject: Re: [PATCH] posix-timers: RCU conversion
Date: Mon, 04 Apr 2011 10:18:50 +0300 [thread overview]
Message-ID: <4D9970DA.1010801@redhat.com> (raw)
In-Reply-To: <BANLkTimHBGd+OLnWFMOm8HdFwgbZNYm8Ng@mail.gmail.com>
On 04/04/2011 06:30 AM, Ben Nagy wrote:
> On Tue, Mar 22, 2011 at 4:20 PM, Avi Kivity<avi@redhat.com> wrote:
> [...]
> > Looks like the posix-timer issue is completely gone, to be replaced by the
> > load balancer.
> >
> > Copying peterz.
>
> Hi all,
>
> I feel bad about such a big cc list, but I don't know who can be left out :/
>
> Still got the performance issue with the load balancer.
>
> Additionally, when trying to test with 48 VMs under load, they
> consistently all lock up after a few minutes, which we traced as far
> as ocfs2 dying. We're investigating it as an ocfs2 issue, and we'll
> try and replicate on ext4 / nfs, but I just thought it might be
> somehow related.
>
> Apr 3 18:27:35 eax kernel: [16029.399507] ------------[ cut here ]------------
> Apr 3 18:27:35 eax kernel: [16029.401408] kernel BUG at
> /home/fuzzadmin/src/natty/source/fs/jbd2/journal.c:1610!
> Apr 3 18:27:35 eax kernel: [16029.404541] invalid opcode: 0000 [#1] SMP
> Apr 3 18:27:35 eax kernel: [16029.406289] last sysfs file:
> /sys/devices/system/cpu/cpu47/cache/index2/shared_cpu_map
> Apr 3 18:27:35 eax kernel: [16029.409453] CPU 36
> Apr 3 18:27:35 eax kernel: [16029.409453] Modules linked in: ocfs2
> quota_tree ip6table_filter ip6_tables ipt_MASQUERADE iptable_nat
> nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntr
> ack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter
> ip_tables x_tables bridge stp w83627ehf hwmon_vid ocfs2_dlmfs
> ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue c
> onfigfs ipmi_si ipmi_msghandler ib_srp scsi_transport_srp scsi_tgt
> vesafb ib_ipoib ib_iser ib_umad iscsi_tcp rdma_ucm libiscsi_tcp
> rdma_cm libiscsi ib_cm iw_cm scsi_transport_iscsi ib_addr ib
> _sa ib_uverbs mlx4_ib ib_mad ib_core vhost_net amd64_edac_mod psmouse
> edac_core sp5100_tco ghes joydev serio_raw hed edac_mce_amd k10temp
> kvm_amd i2c_piix4 kvm lp parport usb_storage usbhid h
> id uas igb ahci pata_atiixp libahci dca mlx4_core
> Apr 3 18:27:35 eax kernel: [16029.409453]
> Apr 3 18:27:35 eax kernel: [16029.409453] Pid: 1746, comm: ocfs2cmt
> Not tainted 2.6.38-7-server #39 Supermicro H8QG6/H8QG6
> Apr 3 18:27:35 eax kernel: [16029.409453] RIP:
> 0010:[<ffffffff8124905a>] [<ffffffff8124905a>]
> jbd2_journal_flush+0x17a/0x190
> Apr 3 18:27:35 eax kernel: [16029.409453] RSP: 0018:ffff880404be3dc0
> EFLAGS: 00010286
> Apr 3 18:27:35 eax kernel: [16029.409453] RAX: 0000000000000029 RBX:
> ffff88181593c000 RCX: 000000000000001e
> Apr 3 18:27:35 eax kernel: [16029.409453] RDX: 00000000fffffffb RSI:
> ffff880404be3cd0 RDI: ffff88181593c024
> Apr 3 18:27:35 eax kernel: [16029.409453] RBP: ffff880404be3df0 R08:
> ffff880404be2000 R09: 0000000000000000
> Apr 3 18:27:35 eax kernel: [16029.409453] R10: 0000000000000000 R11:
> 0000000000000001 R12: 0000000000000e5d
> Apr 3 18:27:35 eax kernel: [16029.409453] R13: ffff88181593c39c R14:
> ffff88181593c024 R15: 0000000000000000
> Apr 3 18:27:35 eax kernel: [16029.409453] FS: 00007f73affb7700(0000)
> GS:ffff881627c00000(0000) knlGS:0000000000000000
> Apr 3 18:27:35 eax kernel: [16029.736049] CS: 0010 DS: 0000 ES: 0000
> CR0: 000000008005003b
> Apr 3 18:27:35 eax kernel: [16029.736049] CR2: 0000000002f72000 CR3:
> 00000002f0d31000 CR4: 00000000000006e0
> Apr 3 18:27:35 eax kernel: [16029.736049] DR0: 00000000000000a0 DR1:
> 0000000000000000 DR2: 0000000000000003
> Apr 3 18:27:35 eax kernel: [16029.736049] DR3: 00000000000000b0 DR6:
> 00000000ffff0ff0 DR7: 0000000000000400
> Apr 3 18:27:35 eax kernel: [16029.736049] Process ocfs2cmt (pid:
> 1746, threadinfo ffff880404be2000, task ffff8803f72396e0)
> Apr 3 18:27:35 eax kernel: [16029.736049] Stack:
> Apr 3 18:27:35 eax kernel: [16029.736049] 0000000000000001
> ffff8806155eb800 ffff8806155eb838 ffff8803f72396e0
> Apr 3 18:27:35 eax kernel: [16029.736049] ffff8806160cb000
> ffff8806160cb160 ffff880404be3e40 ffffffffa042af12
> Apr 3 18:27:35 eax kernel: [16029.736049] 0000000000000000
> ffff8806155eb860 0000000000000286 ffff8806155eb828
> Apr 3 18:27:35 eax kernel: [16029.736049] Call Trace:
> Apr 3 18:27:35 eax kernel: [16029.736049] [<ffffffffa042af12>]
> ocfs2_commit_cache+0xc2/0x330 [ocfs2]
> Apr 3 18:27:35 eax kernel: [16029.736049] [<ffffffffa042b1e1>]
> ocfs2_commit_thread+0x61/0x210 [ocfs2]
> Apr 3 18:27:35 eax kernel: [16029.736049] [<ffffffff81087870>] ?
> autoremove_wake_function+0x0/0x40
> Apr 3 18:27:35 eax kernel: [16029.736049] [<ffffffffa042b180>] ?
> ocfs2_commit_thread+0x0/0x210 [ocfs2]
> Apr 3 18:27:35 eax kernel: [16029.736049] [<ffffffff81087126>]
> kthread+0x96/0xa0
> Apr 3 18:27:35 eax kernel: [16029.736049] [<ffffffff8100cde4>]
> kernel_thread_helper+0x4/0x10
> Apr 3 18:27:35 eax kernel: [16029.736049] [<ffffffff81087090>] ?
> kthread+0x0/0xa0
> Apr 3 18:27:35 eax kernel: [16029.736049] [<ffffffff8100cde0>] ?
> kernel_thread_helper+0x0/0x10
> Apr 3 18:27:35 eax kernel: [16029.736049] Code: c0 5b 41 5c 41 5d 41
> 5e 41 5f c9 c3 0f 1f 44 00 00 4c 8b 63 58 4d 85 e4 0f 85 d2 fe ff ff
> f0 81 43 24 00 00 00 01 e9 da fe ff ff<0f> 0b 0f 0b 0f 0b 0f 0b 0f 0b
> 66 66 66 2e 0f 1f 84 00 00 00 00
> Apr 3 18:27:35 eax kernel: [16029.736049] RIP [<ffffffff8124905a>]
> jbd2_journal_flush+0x17a/0x190
> Apr 3 18:27:35 eax kernel: [16031.320082] ------------[ cut here ]------------
Please report this to the ocfs2 developers.
--
error compiling committee.c: too many arguments to function
WARNING: multiple messages have this Message-ID (diff)
From: Avi Kivity <avi@redhat.com>
To: ben@iagu.net
Cc: Eric Dumazet <eric.dumazet@gmail.com>,
Thomas Gleixner <tglx@linutronix.de>,
KVM list <kvm@vger.kernel.org>,
linux-kernel <linux-kernel@vger.kernel.org>,
John Stultz <johnstul@us.ibm.com>,
Richard Cochran <richard.cochran@omicron.at>,
Peter Zijlstra <a.p.zijlstra@chello.nl>
Subject: Re: [PATCH] posix-timers: RCU conversion
Date: Mon, 04 Apr 2011 10:18:50 +0300 [thread overview]
Message-ID: <4D9970DA.1010801@redhat.com> (raw)
In-Reply-To: <BANLkTimHBGd+OLnWFMOm8HdFwgbZNYm8Ng@mail.gmail.com>
On 04/04/2011 06:30 AM, Ben Nagy wrote:
> On Tue, Mar 22, 2011 at 4:20 PM, Avi Kivity<avi@redhat.com> wrote:
> [...]
> > Looks like the posix-timer issue is completely gone, to be replaced by the
> > load balancer.
> >
> > Copying peterz.
>
> Hi all,
>
> I feel bad about such a big cc list, but I don't know who can be left out :/
>
> Still got the performance issue with the load balancer.
>
> Additionally, when trying to test with 48 VMs under load, they
> consistently all lock up after a few minutes, which we traced as far
> as ocfs2 dying. We're investigating it as an ocfs2 issue, and we'll
> try and replicate on ext4 / nfs, but I just thought it might be
> somehow related.
>
> Apr 3 18:27:35 eax kernel: [16029.399507] ------------[ cut here ]------------
> Apr 3 18:27:35 eax kernel: [16029.401408] kernel BUG at
> /home/fuzzadmin/src/natty/source/fs/jbd2/journal.c:1610!
> Apr 3 18:27:35 eax kernel: [16029.404541] invalid opcode: 0000 [#1] SMP
> Apr 3 18:27:35 eax kernel: [16029.406289] last sysfs file:
> /sys/devices/system/cpu/cpu47/cache/index2/shared_cpu_map
> Apr 3 18:27:35 eax kernel: [16029.409453] CPU 36
> Apr 3 18:27:35 eax kernel: [16029.409453] Modules linked in: ocfs2
> quota_tree ip6table_filter ip6_tables ipt_MASQUERADE iptable_nat
> nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntr
> ack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter
> ip_tables x_tables bridge stp w83627ehf hwmon_vid ocfs2_dlmfs
> ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue c
> onfigfs ipmi_si ipmi_msghandler ib_srp scsi_transport_srp scsi_tgt
> vesafb ib_ipoib ib_iser ib_umad iscsi_tcp rdma_ucm libiscsi_tcp
> rdma_cm libiscsi ib_cm iw_cm scsi_transport_iscsi ib_addr ib
> _sa ib_uverbs mlx4_ib ib_mad ib_core vhost_net amd64_edac_mod psmouse
> edac_core sp5100_tco ghes joydev serio_raw hed edac_mce_amd k10temp
> kvm_amd i2c_piix4 kvm lp parport usb_storage usbhid h
> id uas igb ahci pata_atiixp libahci dca mlx4_core
> Apr 3 18:27:35 eax kernel: [16029.409453]
> Apr 3 18:27:35 eax kernel: [16029.409453] Pid: 1746, comm: ocfs2cmt
> Not tainted 2.6.38-7-server #39 Supermicro H8QG6/H8QG6
> Apr 3 18:27:35 eax kernel: [16029.409453] RIP:
> 0010:[<ffffffff8124905a>] [<ffffffff8124905a>]
> jbd2_journal_flush+0x17a/0x190
> Apr 3 18:27:35 eax kernel: [16029.409453] RSP: 0018:ffff880404be3dc0
> EFLAGS: 00010286
> Apr 3 18:27:35 eax kernel: [16029.409453] RAX: 0000000000000029 RBX:
> ffff88181593c000 RCX: 000000000000001e
> Apr 3 18:27:35 eax kernel: [16029.409453] RDX: 00000000fffffffb RSI:
> ffff880404be3cd0 RDI: ffff88181593c024
> Apr 3 18:27:35 eax kernel: [16029.409453] RBP: ffff880404be3df0 R08:
> ffff880404be2000 R09: 0000000000000000
> Apr 3 18:27:35 eax kernel: [16029.409453] R10: 0000000000000000 R11:
> 0000000000000001 R12: 0000000000000e5d
> Apr 3 18:27:35 eax kernel: [16029.409453] R13: ffff88181593c39c R14:
> ffff88181593c024 R15: 0000000000000000
> Apr 3 18:27:35 eax kernel: [16029.409453] FS: 00007f73affb7700(0000)
> GS:ffff881627c00000(0000) knlGS:0000000000000000
> Apr 3 18:27:35 eax kernel: [16029.736049] CS: 0010 DS: 0000 ES: 0000
> CR0: 000000008005003b
> Apr 3 18:27:35 eax kernel: [16029.736049] CR2: 0000000002f72000 CR3:
> 00000002f0d31000 CR4: 00000000000006e0
> Apr 3 18:27:35 eax kernel: [16029.736049] DR0: 00000000000000a0 DR1:
> 0000000000000000 DR2: 0000000000000003
> Apr 3 18:27:35 eax kernel: [16029.736049] DR3: 00000000000000b0 DR6:
> 00000000ffff0ff0 DR7: 0000000000000400
> Apr 3 18:27:35 eax kernel: [16029.736049] Process ocfs2cmt (pid:
> 1746, threadinfo ffff880404be2000, task ffff8803f72396e0)
> Apr 3 18:27:35 eax kernel: [16029.736049] Stack:
> Apr 3 18:27:35 eax kernel: [16029.736049] 0000000000000001
> ffff8806155eb800 ffff8806155eb838 ffff8803f72396e0
> Apr 3 18:27:35 eax kernel: [16029.736049] ffff8806160cb000
> ffff8806160cb160 ffff880404be3e40 ffffffffa042af12
> Apr 3 18:27:35 eax kernel: [16029.736049] 0000000000000000
> ffff8806155eb860 0000000000000286 ffff8806155eb828
> Apr 3 18:27:35 eax kernel: [16029.736049] Call Trace:
> Apr 3 18:27:35 eax kernel: [16029.736049] [<ffffffffa042af12>]
> ocfs2_commit_cache+0xc2/0x330 [ocfs2]
> Apr 3 18:27:35 eax kernel: [16029.736049] [<ffffffffa042b1e1>]
> ocfs2_commit_thread+0x61/0x210 [ocfs2]
> Apr 3 18:27:35 eax kernel: [16029.736049] [<ffffffff81087870>] ?
> autoremove_wake_function+0x0/0x40
> Apr 3 18:27:35 eax kernel: [16029.736049] [<ffffffffa042b180>] ?
> ocfs2_commit_thread+0x0/0x210 [ocfs2]
> Apr 3 18:27:35 eax kernel: [16029.736049] [<ffffffff81087126>]
> kthread+0x96/0xa0
> Apr 3 18:27:35 eax kernel: [16029.736049] [<ffffffff8100cde4>]
> kernel_thread_helper+0x4/0x10
> Apr 3 18:27:35 eax kernel: [16029.736049] [<ffffffff81087090>] ?
> kthread+0x0/0xa0
> Apr 3 18:27:35 eax kernel: [16029.736049] [<ffffffff8100cde0>] ?
> kernel_thread_helper+0x0/0x10
> Apr 3 18:27:35 eax kernel: [16029.736049] Code: c0 5b 41 5c 41 5d 41
> 5e 41 5f c9 c3 0f 1f 44 00 00 4c 8b 63 58 4d 85 e4 0f 85 d2 fe ff ff
> f0 81 43 24 00 00 00 01 e9 da fe ff ff<0f> 0b 0f 0b 0f 0b 0f 0b 0f 0b
> 66 66 66 2e 0f 1f 84 00 00 00 00
> Apr 3 18:27:35 eax kernel: [16029.736049] RIP [<ffffffff8124905a>]
> jbd2_journal_flush+0x17a/0x190
> Apr 3 18:27:35 eax kernel: [16031.320082] ------------[ cut here ]------------
Please report this to the ocfs2 developers.
--
error compiling committee.c: too many arguments to function
next prev parent reply other threads:[~2011-04-04 7:19 UTC|newest]
Thread overview: 46+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-03-18 12:02 KVM lock contention on 48 core AMD machine Ben Nagy
2011-03-18 12:30 ` Joerg Roedel
2011-03-19 4:45 ` Ben Nagy
2011-03-21 9:50 ` Avi Kivity
2011-03-21 11:43 ` Ben Nagy
2011-03-21 13:41 ` Ben Nagy
2011-03-21 13:53 ` Avi Kivity
[not found] ` <AANLkTikWQS281kTtJ32-qo5U+w_BAak7qUwVhUQgOxxv@mail.gmail.com>
2011-03-21 15:50 ` Avi Kivity
2011-03-21 16:16 ` Ben Nagy
2011-03-21 16:33 ` Avi Kivity
2011-03-21 16:54 ` Eric Dumazet
2011-03-21 17:02 ` Avi Kivity
2011-03-21 17:12 ` Eric Dumazet
2011-03-21 18:12 ` Ben Nagy
2011-03-21 22:27 ` [RFC] posix-timers: RCU conversion Eric Dumazet
2011-03-22 7:09 ` [PATCH] " Eric Dumazet
2011-03-22 8:59 ` Ben Nagy
2011-03-22 10:35 ` Avi Kivity
2011-03-22 10:35 ` Avi Kivity
2011-04-04 3:30 ` Ben Nagy
2011-04-04 7:18 ` Avi Kivity [this message]
2011-04-04 7:18 ` Avi Kivity
2011-04-05 7:49 ` Peter Zijlstra
2011-04-05 8:16 ` Avi Kivity
2011-04-05 8:16 ` Avi Kivity
2011-04-05 8:48 ` Peter Zijlstra
2011-04-05 8:56 ` Avi Kivity
2011-04-05 8:56 ` Avi Kivity
2011-04-05 9:03 ` Peter Zijlstra
2011-04-05 9:08 ` Avi Kivity
2011-04-05 9:08 ` Avi Kivity
2011-04-05 9:50 ` Ben Nagy
2011-04-05 8:56 ` Mike Galbraith
2011-04-03 16:54 ` Eric Dumazet
2011-04-04 18:08 ` john stultz
2011-04-04 19:47 ` Eric Dumazet
2011-04-05 14:48 ` Oleg Nesterov
2011-04-05 15:18 ` [PATCH v2] " Eric Dumazet
2011-04-05 15:43 ` Oleg Nesterov
2011-05-16 15:10 ` Avi Kivity
2011-05-16 15:30 ` Eric Dumazet
2011-04-08 18:28 ` [PATCH] " Paul E. McKenney
2011-03-21 18:14 ` KVM lock contention on 48 core AMD machine Avi Kivity
2011-03-21 18:48 ` Michael Tokarev
2011-03-21 18:53 ` Avi Kivity
2011-03-18 12:44 ` Stefan Hajnoczi
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4D9970DA.1010801@redhat.com \
--to=avi@redhat.com \
--cc=a.p.zijlstra@chello.nl \
--cc=ben@iagu.net \
--cc=eric.dumazet@gmail.com \
--cc=johnstul@us.ibm.com \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=richard.cochran@omicron.at \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.