From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753878Ab1DDHTr (ORCPT ); Mon, 4 Apr 2011 03:19:47 -0400 Received: from mx1.redhat.com ([209.132.183.28]:20565 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751165Ab1DDHTq (ORCPT ); Mon, 4 Apr 2011 03:19:46 -0400 Message-ID: <4D9970DA.1010801@redhat.com> Date: Mon, 04 Apr 2011 10:18:50 +0300 From: Avi Kivity User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110307 Fedora/3.1.9-0.39.b3pre.fc14 Lightning/1.0b3pre Thunderbird/3.1.9 MIME-Version: 1.0 To: ben@iagu.net CC: Eric Dumazet , Thomas Gleixner , KVM list , linux-kernel , John Stultz , Richard Cochran , Peter Zijlstra Subject: Re: [PATCH] posix-timers: RCU conversion References: <20110318123031.GB6066@8bytes.org> <4D871F6C.40207@redhat.com> <4D875842.9050308@redhat.com> <4D8773AA.8030408@redhat.com> <1300726498.2884.493.camel@edumazet-laptop> <4D8784A9.8040303@redhat.com> <1300727545.2884.513.camel@edumazet-laptop> <1300746429.2837.20.camel@edumazet-laptop> <1300777760.2837.38.camel@edumazet-laptop> <4D887B6A.2020102@redhat.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 04/04/2011 06:30 AM, Ben Nagy wrote: > On Tue, Mar 22, 2011 at 4:20 PM, Avi Kivity wrote: > [...] > > Looks like the posix-timer issue is completely gone, to be replaced by the > > load balancer. > > > > Copying peterz. > > Hi all, > > I feel bad about such a big cc list, but I don't know who can be left out :/ > > Still got the performance issue with the load balancer. > > Additionally, when trying to test with 48 VMs under load, they > consistently all lock up after a few minutes, which we traced as far > as ocfs2 dying. We're investigating it as an ocfs2 issue, and we'll > try and replicate on ext4 / nfs, but I just thought it might be > somehow related. > > Apr 3 18:27:35 eax kernel: [16029.399507] ------------[ cut here ]------------ > Apr 3 18:27:35 eax kernel: [16029.401408] kernel BUG at > /home/fuzzadmin/src/natty/source/fs/jbd2/journal.c:1610! > Apr 3 18:27:35 eax kernel: [16029.404541] invalid opcode: 0000 [#1] SMP > Apr 3 18:27:35 eax kernel: [16029.406289] last sysfs file: > /sys/devices/system/cpu/cpu47/cache/index2/shared_cpu_map > Apr 3 18:27:35 eax kernel: [16029.409453] CPU 36 > Apr 3 18:27:35 eax kernel: [16029.409453] Modules linked in: ocfs2 > quota_tree ip6table_filter ip6_tables ipt_MASQUERADE iptable_nat > nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntr > ack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter > ip_tables x_tables bridge stp w83627ehf hwmon_vid ocfs2_dlmfs > ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue c > onfigfs ipmi_si ipmi_msghandler ib_srp scsi_transport_srp scsi_tgt > vesafb ib_ipoib ib_iser ib_umad iscsi_tcp rdma_ucm libiscsi_tcp > rdma_cm libiscsi ib_cm iw_cm scsi_transport_iscsi ib_addr ib > _sa ib_uverbs mlx4_ib ib_mad ib_core vhost_net amd64_edac_mod psmouse > edac_core sp5100_tco ghes joydev serio_raw hed edac_mce_amd k10temp > kvm_amd i2c_piix4 kvm lp parport usb_storage usbhid h > id uas igb ahci pata_atiixp libahci dca mlx4_core > Apr 3 18:27:35 eax kernel: [16029.409453] > Apr 3 18:27:35 eax kernel: [16029.409453] Pid: 1746, comm: ocfs2cmt > Not tainted 2.6.38-7-server #39 Supermicro H8QG6/H8QG6 > Apr 3 18:27:35 eax kernel: [16029.409453] RIP: > 0010:[] [] > jbd2_journal_flush+0x17a/0x190 > Apr 3 18:27:35 eax kernel: [16029.409453] RSP: 0018:ffff880404be3dc0 > EFLAGS: 00010286 > Apr 3 18:27:35 eax kernel: [16029.409453] RAX: 0000000000000029 RBX: > ffff88181593c000 RCX: 000000000000001e > Apr 3 18:27:35 eax kernel: [16029.409453] RDX: 00000000fffffffb RSI: > ffff880404be3cd0 RDI: ffff88181593c024 > Apr 3 18:27:35 eax kernel: [16029.409453] RBP: ffff880404be3df0 R08: > ffff880404be2000 R09: 0000000000000000 > Apr 3 18:27:35 eax kernel: [16029.409453] R10: 0000000000000000 R11: > 0000000000000001 R12: 0000000000000e5d > Apr 3 18:27:35 eax kernel: [16029.409453] R13: ffff88181593c39c R14: > ffff88181593c024 R15: 0000000000000000 > Apr 3 18:27:35 eax kernel: [16029.409453] FS: 00007f73affb7700(0000) > GS:ffff881627c00000(0000) knlGS:0000000000000000 > Apr 3 18:27:35 eax kernel: [16029.736049] CS: 0010 DS: 0000 ES: 0000 > CR0: 000000008005003b > Apr 3 18:27:35 eax kernel: [16029.736049] CR2: 0000000002f72000 CR3: > 00000002f0d31000 CR4: 00000000000006e0 > Apr 3 18:27:35 eax kernel: [16029.736049] DR0: 00000000000000a0 DR1: > 0000000000000000 DR2: 0000000000000003 > Apr 3 18:27:35 eax kernel: [16029.736049] DR3: 00000000000000b0 DR6: > 00000000ffff0ff0 DR7: 0000000000000400 > Apr 3 18:27:35 eax kernel: [16029.736049] Process ocfs2cmt (pid: > 1746, threadinfo ffff880404be2000, task ffff8803f72396e0) > Apr 3 18:27:35 eax kernel: [16029.736049] Stack: > Apr 3 18:27:35 eax kernel: [16029.736049] 0000000000000001 > ffff8806155eb800 ffff8806155eb838 ffff8803f72396e0 > Apr 3 18:27:35 eax kernel: [16029.736049] ffff8806160cb000 > ffff8806160cb160 ffff880404be3e40 ffffffffa042af12 > Apr 3 18:27:35 eax kernel: [16029.736049] 0000000000000000 > ffff8806155eb860 0000000000000286 ffff8806155eb828 > Apr 3 18:27:35 eax kernel: [16029.736049] Call Trace: > Apr 3 18:27:35 eax kernel: [16029.736049] [] > ocfs2_commit_cache+0xc2/0x330 [ocfs2] > Apr 3 18:27:35 eax kernel: [16029.736049] [] > ocfs2_commit_thread+0x61/0x210 [ocfs2] > Apr 3 18:27:35 eax kernel: [16029.736049] [] ? > autoremove_wake_function+0x0/0x40 > Apr 3 18:27:35 eax kernel: [16029.736049] [] ? > ocfs2_commit_thread+0x0/0x210 [ocfs2] > Apr 3 18:27:35 eax kernel: [16029.736049] [] > kthread+0x96/0xa0 > Apr 3 18:27:35 eax kernel: [16029.736049] [] > kernel_thread_helper+0x4/0x10 > Apr 3 18:27:35 eax kernel: [16029.736049] [] ? > kthread+0x0/0xa0 > Apr 3 18:27:35 eax kernel: [16029.736049] [] ? > kernel_thread_helper+0x0/0x10 > Apr 3 18:27:35 eax kernel: [16029.736049] Code: c0 5b 41 5c 41 5d 41 > 5e 41 5f c9 c3 0f 1f 44 00 00 4c 8b 63 58 4d 85 e4 0f 85 d2 fe ff ff > f0 81 43 24 00 00 00 01 e9 da fe ff ff<0f> 0b 0f 0b 0f 0b 0f 0b 0f 0b > 66 66 66 2e 0f 1f 84 00 00 00 00 > Apr 3 18:27:35 eax kernel: [16029.736049] RIP [] > jbd2_journal_flush+0x17a/0x190 > Apr 3 18:27:35 eax kernel: [16031.320082] ------------[ cut here ]------------ Please report this to the ocfs2 developers. -- error compiling committee.c: too many arguments to function From mboxrd@z Thu Jan 1 00:00:00 1970 From: Avi Kivity Subject: Re: [PATCH] posix-timers: RCU conversion Date: Mon, 04 Apr 2011 10:18:50 +0300 Message-ID: <4D9970DA.1010801@redhat.com> References: <20110318123031.GB6066@8bytes.org> <4D871F6C.40207@redhat.com> <4D875842.9050308@redhat.com> <4D8773AA.8030408@redhat.com> <1300726498.2884.493.camel@edumazet-laptop> <4D8784A9.8040303@redhat.com> <1300727545.2884.513.camel@edumazet-laptop> <1300746429.2837.20.camel@edumazet-laptop> <1300777760.2837.38.camel@edumazet-laptop> <4D887B6A.2020102@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Eric Dumazet , Thomas Gleixner , KVM list , linux-kernel , John Stultz , Richard Cochran , Peter Zijlstra To: ben@iagu.net Return-path: In-Reply-To: Sender: linux-kernel-owner@vger.kernel.org List-Id: kvm.vger.kernel.org On 04/04/2011 06:30 AM, Ben Nagy wrote: > On Tue, Mar 22, 2011 at 4:20 PM, Avi Kivity wrote: > [...] > > Looks like the posix-timer issue is completely gone, to be replaced by the > > load balancer. > > > > Copying peterz. > > Hi all, > > I feel bad about such a big cc list, but I don't know who can be left out :/ > > Still got the performance issue with the load balancer. > > Additionally, when trying to test with 48 VMs under load, they > consistently all lock up after a few minutes, which we traced as far > as ocfs2 dying. We're investigating it as an ocfs2 issue, and we'll > try and replicate on ext4 / nfs, but I just thought it might be > somehow related. > > Apr 3 18:27:35 eax kernel: [16029.399507] ------------[ cut here ]------------ > Apr 3 18:27:35 eax kernel: [16029.401408] kernel BUG at > /home/fuzzadmin/src/natty/source/fs/jbd2/journal.c:1610! > Apr 3 18:27:35 eax kernel: [16029.404541] invalid opcode: 0000 [#1] SMP > Apr 3 18:27:35 eax kernel: [16029.406289] last sysfs file: > /sys/devices/system/cpu/cpu47/cache/index2/shared_cpu_map > Apr 3 18:27:35 eax kernel: [16029.409453] CPU 36 > Apr 3 18:27:35 eax kernel: [16029.409453] Modules linked in: ocfs2 > quota_tree ip6table_filter ip6_tables ipt_MASQUERADE iptable_nat > nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntr > ack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter > ip_tables x_tables bridge stp w83627ehf hwmon_vid ocfs2_dlmfs > ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue c > onfigfs ipmi_si ipmi_msghandler ib_srp scsi_transport_srp scsi_tgt > vesafb ib_ipoib ib_iser ib_umad iscsi_tcp rdma_ucm libiscsi_tcp > rdma_cm libiscsi ib_cm iw_cm scsi_transport_iscsi ib_addr ib > _sa ib_uverbs mlx4_ib ib_mad ib_core vhost_net amd64_edac_mod psmouse > edac_core sp5100_tco ghes joydev serio_raw hed edac_mce_amd k10temp > kvm_amd i2c_piix4 kvm lp parport usb_storage usbhid h > id uas igb ahci pata_atiixp libahci dca mlx4_core > Apr 3 18:27:35 eax kernel: [16029.409453] > Apr 3 18:27:35 eax kernel: [16029.409453] Pid: 1746, comm: ocfs2cmt > Not tainted 2.6.38-7-server #39 Supermicro H8QG6/H8QG6 > Apr 3 18:27:35 eax kernel: [16029.409453] RIP: > 0010:[] [] > jbd2_journal_flush+0x17a/0x190 > Apr 3 18:27:35 eax kernel: [16029.409453] RSP: 0018:ffff880404be3dc0 > EFLAGS: 00010286 > Apr 3 18:27:35 eax kernel: [16029.409453] RAX: 0000000000000029 RBX: > ffff88181593c000 RCX: 000000000000001e > Apr 3 18:27:35 eax kernel: [16029.409453] RDX: 00000000fffffffb RSI: > ffff880404be3cd0 RDI: ffff88181593c024 > Apr 3 18:27:35 eax kernel: [16029.409453] RBP: ffff880404be3df0 R08: > ffff880404be2000 R09: 0000000000000000 > Apr 3 18:27:35 eax kernel: [16029.409453] R10: 0000000000000000 R11: > 0000000000000001 R12: 0000000000000e5d > Apr 3 18:27:35 eax kernel: [16029.409453] R13: ffff88181593c39c R14: > ffff88181593c024 R15: 0000000000000000 > Apr 3 18:27:35 eax kernel: [16029.409453] FS: 00007f73affb7700(0000) > GS:ffff881627c00000(0000) knlGS:0000000000000000 > Apr 3 18:27:35 eax kernel: [16029.736049] CS: 0010 DS: 0000 ES: 0000 > CR0: 000000008005003b > Apr 3 18:27:35 eax kernel: [16029.736049] CR2: 0000000002f72000 CR3: > 00000002f0d31000 CR4: 00000000000006e0 > Apr 3 18:27:35 eax kernel: [16029.736049] DR0: 00000000000000a0 DR1: > 0000000000000000 DR2: 0000000000000003 > Apr 3 18:27:35 eax kernel: [16029.736049] DR3: 00000000000000b0 DR6: > 00000000ffff0ff0 DR7: 0000000000000400 > Apr 3 18:27:35 eax kernel: [16029.736049] Process ocfs2cmt (pid: > 1746, threadinfo ffff880404be2000, task ffff8803f72396e0) > Apr 3 18:27:35 eax kernel: [16029.736049] Stack: > Apr 3 18:27:35 eax kernel: [16029.736049] 0000000000000001 > ffff8806155eb800 ffff8806155eb838 ffff8803f72396e0 > Apr 3 18:27:35 eax kernel: [16029.736049] ffff8806160cb000 > ffff8806160cb160 ffff880404be3e40 ffffffffa042af12 > Apr 3 18:27:35 eax kernel: [16029.736049] 0000000000000000 > ffff8806155eb860 0000000000000286 ffff8806155eb828 > Apr 3 18:27:35 eax kernel: [16029.736049] Call Trace: > Apr 3 18:27:35 eax kernel: [16029.736049] [] > ocfs2_commit_cache+0xc2/0x330 [ocfs2] > Apr 3 18:27:35 eax kernel: [16029.736049] [] > ocfs2_commit_thread+0x61/0x210 [ocfs2] > Apr 3 18:27:35 eax kernel: [16029.736049] [] ? > autoremove_wake_function+0x0/0x40 > Apr 3 18:27:35 eax kernel: [16029.736049] [] ? > ocfs2_commit_thread+0x0/0x210 [ocfs2] > Apr 3 18:27:35 eax kernel: [16029.736049] [] > kthread+0x96/0xa0 > Apr 3 18:27:35 eax kernel: [16029.736049] [] > kernel_thread_helper+0x4/0x10 > Apr 3 18:27:35 eax kernel: [16029.736049] [] ? > kthread+0x0/0xa0 > Apr 3 18:27:35 eax kernel: [16029.736049] [] ? > kernel_thread_helper+0x0/0x10 > Apr 3 18:27:35 eax kernel: [16029.736049] Code: c0 5b 41 5c 41 5d 41 > 5e 41 5f c9 c3 0f 1f 44 00 00 4c 8b 63 58 4d 85 e4 0f 85 d2 fe ff ff > f0 81 43 24 00 00 00 01 e9 da fe ff ff<0f> 0b 0f 0b 0f 0b 0f 0b 0f 0b > 66 66 66 2e 0f 1f 84 00 00 00 00 > Apr 3 18:27:35 eax kernel: [16029.736049] RIP [] > jbd2_journal_flush+0x17a/0x190 > Apr 3 18:27:35 eax kernel: [16031.320082] ------------[ cut here ]------------ Please report this to the ocfs2 developers. -- error compiling committee.c: too many arguments to function