* XFS Calltraces by using XFS with Ceph @ 2017-06-14 8:22 list 2017-06-14 12:08 ` Brian Foster 0 siblings, 1 reply; 12+ messages in thread From: list @ 2017-06-14 8:22 UTC (permalink / raw) To: linux-xfs Hello guys, we have currently an issue with our ceph setup based on XFS. Sometimes some nodes are dying with high load with this calltrace in dmesg: [Tue Jun 13 13:18:48 2017] BUG: unable to handle kernel NULL pointer dereference at 00000000000000a0 [Tue Jun 13 13:18:48 2017] IP: [<ffffffffc06555a0>] xfs_da3_node_read+0x30/0xb0 [xfs] [Tue Jun 13 13:18:48 2017] PGD 0 [Tue Jun 13 13:18:48 2017] Oops: 0000 [#1] SMP [Tue Jun 13 13:18:48 2017] Modules linked in: cpuid arc4 md4 nls_utf8 cifs fscache nfnetlink_queue nfnetlink xt_CHECKSUM xt_nat iptable_nat nf_nat_ipv4 xt_NFQUEUE xt_CLASSIFY ip6table_mangle dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag veth dummy bridge stp llc ebtable_filter ebtables iptable_mangle xt_CT iptable_raw nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables xt_tcpudp nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack ip6table_filter ip6_tables x_tables xfs ipmi_devintf dcdbas x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel ipmi_ssif aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac edac_core input_leds joydev lpc_ich ioatdma shpchp 8250_fintek ipmi_si ipmi_msghandler acpi_pad acpi_power_meter [Tue Jun 13 13:18:48 2017] mac_hid vhost_net vhost macvtap macvlan kvm_intel kvm irqbypass cdc_ether nf_nat_ftp tcp_htcp nf_nat_pptp nf_nat_proto_gre nf_conntrack_ftp bonding nf_nat_sip nf_conntrack_sip nf_nat nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack usbnet mii lp parport autofs4 btrfs raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath linear raid10 raid1 hid_generic usbhid hid ixgbe igb vxlan ip6_udp_tunnel ahci dca udp_tunnel libahci i2c_algo_bit ptp megaraid_sas pps_core mdio wmi fjes [Tue Jun 13 13:18:48 2017] CPU: 3 PID: 3844 Comm: tp_fstore_op Not tainted 4.4.0-75-generic #96-Ubuntu [Tue Jun 13 13:18:48 2017] Hardware name: Dell Inc. PowerEdge R720/0XH7F2, BIOS 2.5.4 01/22/2016 [Tue Jun 13 13:18:48 2017] task: ffff881feda65400 ti: ffff883fbda08000 task.ti: ffff883fbda08000 [Tue Jun 13 13:18:48 2017] RIP: 0010:[<ffffffffc06555a0>] [<ffffffffc06555a0>] xfs_da3_node_read+0x30/0xb0 [xfs] [Tue Jun 13 13:18:48 2017] RSP: 0018:ffff883fbda0bc88 EFLAGS: 00010286 [Tue Jun 13 13:18:48 2017] RAX: 0000000000000000 RBX: ffff8801102c5050 RCX: 0000000000000001 [Tue Jun 13 13:18:48 2017] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff883fbda0bc38 [Tue Jun 13 13:18:48 2017] RBP: ffff883fbda0bca8 R08: 0000000000000001 R09: fffffffffffffffe [Tue Jun 13 13:18:48 2017] R10: ffff880007374ae0 R11: 0000000000000001 R12: ffff883fbda0bcd8 [Tue Jun 13 13:18:48 2017] R13: ffff880035ac4c80 R14: 0000000000000001 R15: 000000008b1f4885 [Tue Jun 13 13:18:48 2017] FS: 00007fc574607700(0000) GS:ffff883fff040000(0000) knlGS:0000000000000000 [Tue Jun 13 13:18:48 2017] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Tue Jun 13 13:18:48 2017] CR2: 00000000000000a0 CR3: 0000003fd828d000 CR4: 00000000001426e0 [Tue Jun 13 13:18:48 2017] Stack: [Tue Jun 13 13:18:48 2017] ffffffffc06b4b50 ffffffffc0695ecc ffff883fbda0bde0 0000000000000001 [Tue Jun 13 13:18:48 2017] ffff883fbda0bd20 ffffffffc06718b3 0000000300000008 ffff880e99b44010 [Tue Jun 13 13:18:48 2017] 00000000360c65a8 ffff88270f80b900 0000000000000000 0000000000000000 [Tue Jun 13 13:18:48 2017] Call Trace: [Tue Jun 13 13:18:48 2017] [<ffffffffc0695ecc>] ? xfs_trans_roll+0x2c/0x50 [xfs] [Tue Jun 13 13:18:48 2017] [<ffffffffc06718b3>] xfs_attr3_node_inactive+0x183/0x220 [xfs] [Tue Jun 13 13:18:48 2017] [<ffffffffc06718f9>] xfs_attr3_node_inactive+0x1c9/0x220 [xfs] [Tue Jun 13 13:18:48 2017] [<ffffffffc06719fc>] xfs_attr3_root_inactive+0xac/0x100 [xfs] [Tue Jun 13 13:18:48 2017] [<ffffffffc0671b9c>] xfs_attr_inactive+0x14c/0x1a0 [xfs] [Tue Jun 13 13:18:48 2017] [<ffffffffc068bda5>] xfs_inactive+0x85/0x120 [xfs] [Tue Jun 13 13:18:48 2017] [<ffffffffc06912f5>] xfs_fs_evict_inode+0xa5/0x100 [xfs] [Tue Jun 13 13:18:48 2017] [<ffffffff8122a90e>] evict+0xbe/0x190 [Tue Jun 13 13:18:48 2017] [<ffffffff8122abf1>] iput+0x1c1/0x240 [Tue Jun 13 13:18:48 2017] [<ffffffff8121f6b9>] do_unlinkat+0x199/0x2d0 [Tue Jun 13 13:18:48 2017] [<ffffffff81220256>] SyS_unlink+0x16/0x20 [Tue Jun 13 13:18:48 2017] [<ffffffff8183b972>] entry_SYSCALL_64_fastpath+0x16/0x71 [Tue Jun 13 13:18:48 2017] Code: 55 48 89 e5 41 54 53 4d 89 c4 48 89 fb 48 83 ec 10 48 c7 04 24 50 4b 6b c0 e8 dd fe ff ff 85 c0 75 46 48 85 db 74 41 49 8b 34 24 <48> 8b 96 a0 00 00 00 0f b7 52 08 66 c1 c2 08 66 81 fa be 3e 74 [Tue Jun 13 13:18:48 2017] RIP [<ffffffffc06555a0>] xfs_da3_node_read+0x30/0xb0 [xfs] [Tue Jun 13 13:18:48 2017] RSP <ffff883fbda0bc88> [Tue Jun 13 13:18:48 2017] CR2: 00000000000000a0 [Tue Jun 13 13:18:48 2017] ---[ end trace 5470d0d55cacb4ef ]--- The ceph OSD running on this server has then the issue that it can not reach any other osd in the pool. -1043> 2017-06-13 13:24:00.917597 7fc539a72700 0 -- 192.168.14.19:6827/3389 >> 192.168.14.7:6805/3658 pipe(0x558219846000 sd=23 :6827 s=0 pgs=0 cs=0 l=0 c=0x55821a330400).accept connect_seq 7 vs existing 7 state standby -1042> 2017-06-13 13:24:00.918433 7fc539a72700 0 -- 192.168.14.19:6827/3389 >> 192.168.14.7:6805/3658 pipe(0x558219846000 sd=23 :6827 s=0 pgs=0 cs=0 l=0 c=0x55821a330400).accept connect_seq 8 vs existing 7 state standby -1041> 2017-06-13 13:24:03.654983 7fc4dd21d700 0 -- 192.168.14.19:6825/3389 >> :/0 pipe(0x5581fa6ba000 sd=524 :6825 s=0 pgs=0 cs=0 l=0 c=0x55820a9e5000).accept failed to getpeername (107) Transport endpoint is not connected There are a lot more of these messages. Does any of you have the same issue? We are running Ubuntu 16.04 with kernel 4.4.0-75.96. Best regards, Jonas ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: XFS Calltraces by using XFS with Ceph 2017-06-14 8:22 XFS Calltraces by using XFS with Ceph list @ 2017-06-14 12:08 ` Brian Foster 2017-06-14 13:22 ` list 0 siblings, 1 reply; 12+ messages in thread From: Brian Foster @ 2017-06-14 12:08 UTC (permalink / raw) To: list; +Cc: linux-xfs On Wed, Jun 14, 2017 at 10:22:38AM +0200, list@jonas-server.de wrote: > Hello guys, > > we have currently an issue with our ceph setup based on XFS. Sometimes some > nodes are dying with high load with this calltrace in dmesg: > > [Tue Jun 13 13:18:48 2017] BUG: unable to handle kernel NULL pointer > dereference at 00000000000000a0 > [Tue Jun 13 13:18:48 2017] IP: [<ffffffffc06555a0>] > xfs_da3_node_read+0x30/0xb0 [xfs] > [Tue Jun 13 13:18:48 2017] PGD 0 > [Tue Jun 13 13:18:48 2017] Oops: 0000 [#1] SMP > [Tue Jun 13 13:18:48 2017] Modules linked in: cpuid arc4 md4 nls_utf8 cifs > fscache nfnetlink_queue nfnetlink xt_CHECKSUM xt_nat iptable_nat nf_nat_ipv4 > xt_NFQUEUE xt_CLASSIFY ip6table_mangle dccp_diag dccp tcp_diag udp_diag > inet_diag unix_diag af_packet_diag netlink_diag veth dummy bridge stp llc > ebtable_filter ebtables iptable_mangle xt_CT iptable_raw nf_conntrack_ipv4 > nf_defrag_ipv4 iptable_filter ip_tables xt_tcpudp nf_conntrack_ipv6 > nf_defrag_ipv6 xt_conntrack ip6table_filter ip6_tables x_tables xfs > ipmi_devintf dcdbas x86_pkg_temp_thermal intel_powerclamp coretemp > crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel ipmi_ssif > aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac edac_core > input_leds joydev lpc_ich ioatdma shpchp 8250_fintek ipmi_si ipmi_msghandler > acpi_pad acpi_power_meter > [Tue Jun 13 13:18:48 2017] mac_hid vhost_net vhost macvtap macvlan > kvm_intel kvm irqbypass cdc_ether nf_nat_ftp tcp_htcp nf_nat_pptp > nf_nat_proto_gre nf_conntrack_ftp bonding nf_nat_sip nf_conntrack_sip nf_nat > nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack usbnet mii lp parport > autofs4 btrfs raid456 async_raid6_recov async_memcpy async_pq async_xor > async_tx xor raid6_pq libcrc32c raid0 multipath linear raid10 raid1 > hid_generic usbhid hid ixgbe igb vxlan ip6_udp_tunnel ahci dca udp_tunnel > libahci i2c_algo_bit ptp megaraid_sas pps_core mdio wmi fjes > [Tue Jun 13 13:18:48 2017] CPU: 3 PID: 3844 Comm: tp_fstore_op Not tainted > 4.4.0-75-generic #96-Ubuntu > [Tue Jun 13 13:18:48 2017] Hardware name: Dell Inc. PowerEdge R720/0XH7F2, > BIOS 2.5.4 01/22/2016 > [Tue Jun 13 13:18:48 2017] task: ffff881feda65400 ti: ffff883fbda08000 > task.ti: ffff883fbda08000 > [Tue Jun 13 13:18:48 2017] RIP: 0010:[<ffffffffc06555a0>] > [<ffffffffc06555a0>] xfs_da3_node_read+0x30/0xb0 [xfs] What line does this point at (i.e., 'list *xfs_da3_node_read+0x30' from gdb) on your kernel? Brian > [Tue Jun 13 13:18:48 2017] RSP: 0018:ffff883fbda0bc88 EFLAGS: 00010286 > [Tue Jun 13 13:18:48 2017] RAX: 0000000000000000 RBX: ffff8801102c5050 RCX: > 0000000000000001 > [Tue Jun 13 13:18:48 2017] RDX: 0000000000000000 RSI: 0000000000000000 RDI: > ffff883fbda0bc38 > [Tue Jun 13 13:18:48 2017] RBP: ffff883fbda0bca8 R08: 0000000000000001 R09: > fffffffffffffffe > [Tue Jun 13 13:18:48 2017] R10: ffff880007374ae0 R11: 0000000000000001 R12: > ffff883fbda0bcd8 > [Tue Jun 13 13:18:48 2017] R13: ffff880035ac4c80 R14: 0000000000000001 R15: > 000000008b1f4885 > [Tue Jun 13 13:18:48 2017] FS: 00007fc574607700(0000) > GS:ffff883fff040000(0000) knlGS:0000000000000000 > [Tue Jun 13 13:18:48 2017] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [Tue Jun 13 13:18:48 2017] CR2: 00000000000000a0 CR3: 0000003fd828d000 CR4: > 00000000001426e0 > [Tue Jun 13 13:18:48 2017] Stack: > [Tue Jun 13 13:18:48 2017] ffffffffc06b4b50 ffffffffc0695ecc > ffff883fbda0bde0 0000000000000001 > [Tue Jun 13 13:18:48 2017] ffff883fbda0bd20 ffffffffc06718b3 > 0000000300000008 ffff880e99b44010 > [Tue Jun 13 13:18:48 2017] 00000000360c65a8 ffff88270f80b900 > 0000000000000000 0000000000000000 > [Tue Jun 13 13:18:48 2017] Call Trace: > [Tue Jun 13 13:18:48 2017] [<ffffffffc0695ecc>] ? xfs_trans_roll+0x2c/0x50 > [xfs] > [Tue Jun 13 13:18:48 2017] [<ffffffffc06718b3>] > xfs_attr3_node_inactive+0x183/0x220 [xfs] > [Tue Jun 13 13:18:48 2017] [<ffffffffc06718f9>] > xfs_attr3_node_inactive+0x1c9/0x220 [xfs] > [Tue Jun 13 13:18:48 2017] [<ffffffffc06719fc>] > xfs_attr3_root_inactive+0xac/0x100 [xfs] > [Tue Jun 13 13:18:48 2017] [<ffffffffc0671b9c>] > xfs_attr_inactive+0x14c/0x1a0 [xfs] > [Tue Jun 13 13:18:48 2017] [<ffffffffc068bda5>] xfs_inactive+0x85/0x120 > [xfs] > [Tue Jun 13 13:18:48 2017] [<ffffffffc06912f5>] > xfs_fs_evict_inode+0xa5/0x100 [xfs] > [Tue Jun 13 13:18:48 2017] [<ffffffff8122a90e>] evict+0xbe/0x190 > [Tue Jun 13 13:18:48 2017] [<ffffffff8122abf1>] iput+0x1c1/0x240 > [Tue Jun 13 13:18:48 2017] [<ffffffff8121f6b9>] do_unlinkat+0x199/0x2d0 > [Tue Jun 13 13:18:48 2017] [<ffffffff81220256>] SyS_unlink+0x16/0x20 > [Tue Jun 13 13:18:48 2017] [<ffffffff8183b972>] > entry_SYSCALL_64_fastpath+0x16/0x71 > [Tue Jun 13 13:18:48 2017] Code: 55 48 89 e5 41 54 53 4d 89 c4 48 89 fb 48 > 83 ec 10 48 c7 04 24 50 4b 6b c0 e8 dd fe ff ff 85 c0 75 46 48 85 db 74 41 > 49 8b 34 24 <48> 8b 96 a0 00 00 00 0f b7 52 08 66 c1 c2 08 66 81 fa be 3e 74 > [Tue Jun 13 13:18:48 2017] RIP [<ffffffffc06555a0>] > xfs_da3_node_read+0x30/0xb0 [xfs] > [Tue Jun 13 13:18:48 2017] RSP <ffff883fbda0bc88> > [Tue Jun 13 13:18:48 2017] CR2: 00000000000000a0 > [Tue Jun 13 13:18:48 2017] ---[ end trace 5470d0d55cacb4ef ]--- > > The ceph OSD running on this server has then the issue that it can not reach > any other osd in the pool. > > -1043> 2017-06-13 13:24:00.917597 7fc539a72700 0 -- > 192.168.14.19:6827/3389 >> 192.168.14.7:6805/3658 pipe(0x558219846000 sd=23 > :6827 > s=0 pgs=0 cs=0 l=0 c=0x55821a330400).accept connect_seq 7 vs existing 7 > state standby > -1042> 2017-06-13 13:24:00.918433 7fc539a72700 0 -- > 192.168.14.19:6827/3389 >> 192.168.14.7:6805/3658 pipe(0x558219846000 sd=23 > :6827 > s=0 pgs=0 cs=0 l=0 c=0x55821a330400).accept connect_seq 8 vs existing 7 > state standby > -1041> 2017-06-13 13:24:03.654983 7fc4dd21d700 0 -- > 192.168.14.19:6825/3389 >> :/0 pipe(0x5581fa6ba000 sd=524 :6825 s=0 pgs=0 > cs=0 l=0 > c=0x55820a9e5000).accept failed to getpeername (107) Transport endpoint is > not connected > > > There are a lot more of these messages. Does any of you have the same issue? > We are running Ubuntu 16.04 with kernel 4.4.0-75.96. > > Best regards, > Jonas > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: XFS Calltraces by using XFS with Ceph 2017-06-14 12:08 ` Brian Foster @ 2017-06-14 13:22 ` list 2017-06-14 14:07 ` Brian Foster 0 siblings, 1 reply; 12+ messages in thread From: list @ 2017-06-14 13:22 UTC (permalink / raw) To: Brian Foster; +Cc: linux-xfs I get this output of gdb: # gdb /usr/lib/debug/lib/modules/4.4.0-75-generic/kernel/fs/xfs/xfs.ko GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1 [...] Reading symbols from /usr/lib/debug/lib/modules/4.4.0-75-generic/kernel/fs/xfs/xfs.ko...done. (gdb) list *xfs_da3_node_read+0x30 0x2b5d0 is in xfs_da3_node_read (/build/linux-Hlembm/linux-4.4.0/fs/xfs/libxfs/xfs_da_btree.c:270). 265 /build/linux-Hlembm/linux-4.4.0/fs/xfs/libxfs/xfs_da_btree.c: No such file or directory. Am 2017-06-14 14:08, schrieb Brian Foster: > On Wed, Jun 14, 2017 at 10:22:38AM +0200, list@jonas-server.de wrote: >> Hello guys, >> >> we have currently an issue with our ceph setup based on XFS. Sometimes >> some >> nodes are dying with high load with this calltrace in dmesg: >> >> [Tue Jun 13 13:18:48 2017] BUG: unable to handle kernel NULL pointer >> dereference at 00000000000000a0 >> [Tue Jun 13 13:18:48 2017] IP: [<ffffffffc06555a0>] >> xfs_da3_node_read+0x30/0xb0 [xfs] >> [Tue Jun 13 13:18:48 2017] PGD 0 >> [Tue Jun 13 13:18:48 2017] Oops: 0000 [#1] SMP >> [Tue Jun 13 13:18:48 2017] Modules linked in: cpuid arc4 md4 nls_utf8 >> cifs >> fscache nfnetlink_queue nfnetlink xt_CHECKSUM xt_nat iptable_nat >> nf_nat_ipv4 >> xt_NFQUEUE xt_CLASSIFY ip6table_mangle dccp_diag dccp tcp_diag >> udp_diag >> inet_diag unix_diag af_packet_diag netlink_diag veth dummy bridge stp >> llc >> ebtable_filter ebtables iptable_mangle xt_CT iptable_raw >> nf_conntrack_ipv4 >> nf_defrag_ipv4 iptable_filter ip_tables xt_tcpudp nf_conntrack_ipv6 >> nf_defrag_ipv6 xt_conntrack ip6table_filter ip6_tables x_tables xfs >> ipmi_devintf dcdbas x86_pkg_temp_thermal intel_powerclamp coretemp >> crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel >> ipmi_ssif >> aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac >> edac_core >> input_leds joydev lpc_ich ioatdma shpchp 8250_fintek ipmi_si >> ipmi_msghandler >> acpi_pad acpi_power_meter >> [Tue Jun 13 13:18:48 2017] mac_hid vhost_net vhost macvtap macvlan >> kvm_intel kvm irqbypass cdc_ether nf_nat_ftp tcp_htcp nf_nat_pptp >> nf_nat_proto_gre nf_conntrack_ftp bonding nf_nat_sip nf_conntrack_sip >> nf_nat >> nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack usbnet mii lp >> parport >> autofs4 btrfs raid456 async_raid6_recov async_memcpy async_pq >> async_xor >> async_tx xor raid6_pq libcrc32c raid0 multipath linear raid10 raid1 >> hid_generic usbhid hid ixgbe igb vxlan ip6_udp_tunnel ahci dca >> udp_tunnel >> libahci i2c_algo_bit ptp megaraid_sas pps_core mdio wmi fjes >> [Tue Jun 13 13:18:48 2017] CPU: 3 PID: 3844 Comm: tp_fstore_op Not >> tainted >> 4.4.0-75-generic #96-Ubuntu >> [Tue Jun 13 13:18:48 2017] Hardware name: Dell Inc. PowerEdge >> R720/0XH7F2, >> BIOS 2.5.4 01/22/2016 >> [Tue Jun 13 13:18:48 2017] task: ffff881feda65400 ti: ffff883fbda08000 >> task.ti: ffff883fbda08000 >> [Tue Jun 13 13:18:48 2017] RIP: 0010:[<ffffffffc06555a0>] >> [<ffffffffc06555a0>] xfs_da3_node_read+0x30/0xb0 [xfs] > > What line does this point at (i.e., 'list *xfs_da3_node_read+0x30' from > gdb) on your kernel? > > Brian > >> [Tue Jun 13 13:18:48 2017] RSP: 0018:ffff883fbda0bc88 EFLAGS: >> 00010286 >> [Tue Jun 13 13:18:48 2017] RAX: 0000000000000000 RBX: ffff8801102c5050 >> RCX: >> 0000000000000001 >> [Tue Jun 13 13:18:48 2017] RDX: 0000000000000000 RSI: 0000000000000000 >> RDI: >> ffff883fbda0bc38 >> [Tue Jun 13 13:18:48 2017] RBP: ffff883fbda0bca8 R08: 0000000000000001 >> R09: >> fffffffffffffffe >> [Tue Jun 13 13:18:48 2017] R10: ffff880007374ae0 R11: 0000000000000001 >> R12: >> ffff883fbda0bcd8 >> [Tue Jun 13 13:18:48 2017] R13: ffff880035ac4c80 R14: 0000000000000001 >> R15: >> 000000008b1f4885 >> [Tue Jun 13 13:18:48 2017] FS: 00007fc574607700(0000) >> GS:ffff883fff040000(0000) knlGS:0000000000000000 >> [Tue Jun 13 13:18:48 2017] CS: 0010 DS: 0000 ES: 0000 CR0: >> 0000000080050033 >> [Tue Jun 13 13:18:48 2017] CR2: 00000000000000a0 CR3: 0000003fd828d000 >> CR4: >> 00000000001426e0 >> [Tue Jun 13 13:18:48 2017] Stack: >> [Tue Jun 13 13:18:48 2017] ffffffffc06b4b50 ffffffffc0695ecc >> ffff883fbda0bde0 0000000000000001 >> [Tue Jun 13 13:18:48 2017] ffff883fbda0bd20 ffffffffc06718b3 >> 0000000300000008 ffff880e99b44010 >> [Tue Jun 13 13:18:48 2017] 00000000360c65a8 ffff88270f80b900 >> 0000000000000000 0000000000000000 >> [Tue Jun 13 13:18:48 2017] Call Trace: >> [Tue Jun 13 13:18:48 2017] [<ffffffffc0695ecc>] ? >> xfs_trans_roll+0x2c/0x50 >> [xfs] >> [Tue Jun 13 13:18:48 2017] [<ffffffffc06718b3>] >> xfs_attr3_node_inactive+0x183/0x220 [xfs] >> [Tue Jun 13 13:18:48 2017] [<ffffffffc06718f9>] >> xfs_attr3_node_inactive+0x1c9/0x220 [xfs] >> [Tue Jun 13 13:18:48 2017] [<ffffffffc06719fc>] >> xfs_attr3_root_inactive+0xac/0x100 [xfs] >> [Tue Jun 13 13:18:48 2017] [<ffffffffc0671b9c>] >> xfs_attr_inactive+0x14c/0x1a0 [xfs] >> [Tue Jun 13 13:18:48 2017] [<ffffffffc068bda5>] >> xfs_inactive+0x85/0x120 >> [xfs] >> [Tue Jun 13 13:18:48 2017] [<ffffffffc06912f5>] >> xfs_fs_evict_inode+0xa5/0x100 [xfs] >> [Tue Jun 13 13:18:48 2017] [<ffffffff8122a90e>] evict+0xbe/0x190 >> [Tue Jun 13 13:18:48 2017] [<ffffffff8122abf1>] iput+0x1c1/0x240 >> [Tue Jun 13 13:18:48 2017] [<ffffffff8121f6b9>] >> do_unlinkat+0x199/0x2d0 >> [Tue Jun 13 13:18:48 2017] [<ffffffff81220256>] SyS_unlink+0x16/0x20 >> [Tue Jun 13 13:18:48 2017] [<ffffffff8183b972>] >> entry_SYSCALL_64_fastpath+0x16/0x71 >> [Tue Jun 13 13:18:48 2017] Code: 55 48 89 e5 41 54 53 4d 89 c4 48 89 >> fb 48 >> 83 ec 10 48 c7 04 24 50 4b 6b c0 e8 dd fe ff ff 85 c0 75 46 48 85 db >> 74 41 >> 49 8b 34 24 <48> 8b 96 a0 00 00 00 0f b7 52 08 66 c1 c2 08 66 81 fa be >> 3e 74 >> [Tue Jun 13 13:18:48 2017] RIP [<ffffffffc06555a0>] >> xfs_da3_node_read+0x30/0xb0 [xfs] >> [Tue Jun 13 13:18:48 2017] RSP <ffff883fbda0bc88> >> [Tue Jun 13 13:18:48 2017] CR2: 00000000000000a0 >> [Tue Jun 13 13:18:48 2017] ---[ end trace 5470d0d55cacb4ef ]--- >> >> The ceph OSD running on this server has then the issue that it can not >> reach >> any other osd in the pool. >> >> -1043> 2017-06-13 13:24:00.917597 7fc539a72700 0 -- >> 192.168.14.19:6827/3389 >> 192.168.14.7:6805/3658 pipe(0x558219846000 >> sd=23 >> :6827 >> s=0 pgs=0 cs=0 l=0 c=0x55821a330400).accept connect_seq 7 vs existing >> 7 >> state standby >> -1042> 2017-06-13 13:24:00.918433 7fc539a72700 0 -- >> 192.168.14.19:6827/3389 >> 192.168.14.7:6805/3658 pipe(0x558219846000 >> sd=23 >> :6827 >> s=0 pgs=0 cs=0 l=0 c=0x55821a330400).accept connect_seq 8 vs existing >> 7 >> state standby >> -1041> 2017-06-13 13:24:03.654983 7fc4dd21d700 0 -- >> 192.168.14.19:6825/3389 >> :/0 pipe(0x5581fa6ba000 sd=524 :6825 s=0 >> pgs=0 >> cs=0 l=0 >> c=0x55820a9e5000).accept failed to getpeername (107) Transport >> endpoint is >> not connected >> >> >> There are a lot more of these messages. Does any of you have the same >> issue? >> We are running Ubuntu 16.04 with kernel 4.4.0-75.96. >> >> Best regards, >> Jonas >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" >> in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: XFS Calltraces by using XFS with Ceph 2017-06-14 13:22 ` list @ 2017-06-14 14:07 ` Brian Foster 2017-06-14 15:55 ` Darrick J. Wong 0 siblings, 1 reply; 12+ messages in thread From: Brian Foster @ 2017-06-14 14:07 UTC (permalink / raw) To: list; +Cc: linux-xfs On Wed, Jun 14, 2017 at 03:22:36PM +0200, list@jonas-server.de wrote: > I get this output of gdb: > > # gdb /usr/lib/debug/lib/modules/4.4.0-75-generic/kernel/fs/xfs/xfs.ko > GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1 > [...] > Reading symbols from > /usr/lib/debug/lib/modules/4.4.0-75-generic/kernel/fs/xfs/xfs.ko...done. > (gdb) list *xfs_da3_node_read+0x30 > 0x2b5d0 is in xfs_da3_node_read > (/build/linux-Hlembm/linux-4.4.0/fs/xfs/libxfs/xfs_da_btree.c:270). > 265 /build/linux-Hlembm/linux-4.4.0/fs/xfs/libxfs/xfs_da_btree.c: No such > file or directory. > This would be more helpful if you had the source code available. :P Anyways, I suspect you have a NULL buffer (though I'm not sure where the 0xa0 offset comes from). There have been a couple fixes in that area that come to mind, but it looks to me that v4.4 kernels should already have them. Otherwise, this doesn't ring any bells for me. Perhaps somebody else can chime in on that. I suppose the best next step is to try a more recent, non-distro kernel. If the problem still occurs, see if you can provide a crash dump for analysis. Brian > Am 2017-06-14 14:08, schrieb Brian Foster: > > On Wed, Jun 14, 2017 at 10:22:38AM +0200, list@jonas-server.de wrote: > > > Hello guys, > > > > > > we have currently an issue with our ceph setup based on XFS. > > > Sometimes some > > > nodes are dying with high load with this calltrace in dmesg: > > > > > > [Tue Jun 13 13:18:48 2017] BUG: unable to handle kernel NULL pointer > > > dereference at 00000000000000a0 > > > [Tue Jun 13 13:18:48 2017] IP: [<ffffffffc06555a0>] > > > xfs_da3_node_read+0x30/0xb0 [xfs] > > > [Tue Jun 13 13:18:48 2017] PGD 0 > > > [Tue Jun 13 13:18:48 2017] Oops: 0000 [#1] SMP > > > [Tue Jun 13 13:18:48 2017] Modules linked in: cpuid arc4 md4 > > > nls_utf8 cifs > > > fscache nfnetlink_queue nfnetlink xt_CHECKSUM xt_nat iptable_nat > > > nf_nat_ipv4 > > > xt_NFQUEUE xt_CLASSIFY ip6table_mangle dccp_diag dccp tcp_diag > > > udp_diag > > > inet_diag unix_diag af_packet_diag netlink_diag veth dummy bridge > > > stp llc > > > ebtable_filter ebtables iptable_mangle xt_CT iptable_raw > > > nf_conntrack_ipv4 > > > nf_defrag_ipv4 iptable_filter ip_tables xt_tcpudp nf_conntrack_ipv6 > > > nf_defrag_ipv6 xt_conntrack ip6table_filter ip6_tables x_tables xfs > > > ipmi_devintf dcdbas x86_pkg_temp_thermal intel_powerclamp coretemp > > > crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel > > > ipmi_ssif > > > aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac > > > edac_core > > > input_leds joydev lpc_ich ioatdma shpchp 8250_fintek ipmi_si > > > ipmi_msghandler > > > acpi_pad acpi_power_meter > > > [Tue Jun 13 13:18:48 2017] mac_hid vhost_net vhost macvtap macvlan > > > kvm_intel kvm irqbypass cdc_ether nf_nat_ftp tcp_htcp nf_nat_pptp > > > nf_nat_proto_gre nf_conntrack_ftp bonding nf_nat_sip > > > nf_conntrack_sip nf_nat > > > nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack usbnet mii lp > > > parport > > > autofs4 btrfs raid456 async_raid6_recov async_memcpy async_pq > > > async_xor > > > async_tx xor raid6_pq libcrc32c raid0 multipath linear raid10 raid1 > > > hid_generic usbhid hid ixgbe igb vxlan ip6_udp_tunnel ahci dca > > > udp_tunnel > > > libahci i2c_algo_bit ptp megaraid_sas pps_core mdio wmi fjes > > > [Tue Jun 13 13:18:48 2017] CPU: 3 PID: 3844 Comm: tp_fstore_op Not > > > tainted > > > 4.4.0-75-generic #96-Ubuntu > > > [Tue Jun 13 13:18:48 2017] Hardware name: Dell Inc. PowerEdge > > > R720/0XH7F2, > > > BIOS 2.5.4 01/22/2016 > > > [Tue Jun 13 13:18:48 2017] task: ffff881feda65400 ti: ffff883fbda08000 > > > task.ti: ffff883fbda08000 > > > [Tue Jun 13 13:18:48 2017] RIP: 0010:[<ffffffffc06555a0>] > > > [<ffffffffc06555a0>] xfs_da3_node_read+0x30/0xb0 [xfs] > > > > What line does this point at (i.e., 'list *xfs_da3_node_read+0x30' from > > gdb) on your kernel? > > > > Brian > > > > > [Tue Jun 13 13:18:48 2017] RSP: 0018:ffff883fbda0bc88 EFLAGS: > > > 00010286 > > > [Tue Jun 13 13:18:48 2017] RAX: 0000000000000000 RBX: > > > ffff8801102c5050 RCX: > > > 0000000000000001 > > > [Tue Jun 13 13:18:48 2017] RDX: 0000000000000000 RSI: > > > 0000000000000000 RDI: > > > ffff883fbda0bc38 > > > [Tue Jun 13 13:18:48 2017] RBP: ffff883fbda0bca8 R08: > > > 0000000000000001 R09: > > > fffffffffffffffe > > > [Tue Jun 13 13:18:48 2017] R10: ffff880007374ae0 R11: > > > 0000000000000001 R12: > > > ffff883fbda0bcd8 > > > [Tue Jun 13 13:18:48 2017] R13: ffff880035ac4c80 R14: > > > 0000000000000001 R15: > > > 000000008b1f4885 > > > [Tue Jun 13 13:18:48 2017] FS: 00007fc574607700(0000) > > > GS:ffff883fff040000(0000) knlGS:0000000000000000 > > > [Tue Jun 13 13:18:48 2017] CS: 0010 DS: 0000 ES: 0000 CR0: > > > 0000000080050033 > > > [Tue Jun 13 13:18:48 2017] CR2: 00000000000000a0 CR3: > > > 0000003fd828d000 CR4: > > > 00000000001426e0 > > > [Tue Jun 13 13:18:48 2017] Stack: > > > [Tue Jun 13 13:18:48 2017] ffffffffc06b4b50 ffffffffc0695ecc > > > ffff883fbda0bde0 0000000000000001 > > > [Tue Jun 13 13:18:48 2017] ffff883fbda0bd20 ffffffffc06718b3 > > > 0000000300000008 ffff880e99b44010 > > > [Tue Jun 13 13:18:48 2017] 00000000360c65a8 ffff88270f80b900 > > > 0000000000000000 0000000000000000 > > > [Tue Jun 13 13:18:48 2017] Call Trace: > > > [Tue Jun 13 13:18:48 2017] [<ffffffffc0695ecc>] ? > > > xfs_trans_roll+0x2c/0x50 > > > [xfs] > > > [Tue Jun 13 13:18:48 2017] [<ffffffffc06718b3>] > > > xfs_attr3_node_inactive+0x183/0x220 [xfs] > > > [Tue Jun 13 13:18:48 2017] [<ffffffffc06718f9>] > > > xfs_attr3_node_inactive+0x1c9/0x220 [xfs] > > > [Tue Jun 13 13:18:48 2017] [<ffffffffc06719fc>] > > > xfs_attr3_root_inactive+0xac/0x100 [xfs] > > > [Tue Jun 13 13:18:48 2017] [<ffffffffc0671b9c>] > > > xfs_attr_inactive+0x14c/0x1a0 [xfs] > > > [Tue Jun 13 13:18:48 2017] [<ffffffffc068bda5>] > > > xfs_inactive+0x85/0x120 > > > [xfs] > > > [Tue Jun 13 13:18:48 2017] [<ffffffffc06912f5>] > > > xfs_fs_evict_inode+0xa5/0x100 [xfs] > > > [Tue Jun 13 13:18:48 2017] [<ffffffff8122a90e>] evict+0xbe/0x190 > > > [Tue Jun 13 13:18:48 2017] [<ffffffff8122abf1>] iput+0x1c1/0x240 > > > [Tue Jun 13 13:18:48 2017] [<ffffffff8121f6b9>] > > > do_unlinkat+0x199/0x2d0 > > > [Tue Jun 13 13:18:48 2017] [<ffffffff81220256>] SyS_unlink+0x16/0x20 > > > [Tue Jun 13 13:18:48 2017] [<ffffffff8183b972>] > > > entry_SYSCALL_64_fastpath+0x16/0x71 > > > [Tue Jun 13 13:18:48 2017] Code: 55 48 89 e5 41 54 53 4d 89 c4 48 89 > > > fb 48 > > > 83 ec 10 48 c7 04 24 50 4b 6b c0 e8 dd fe ff ff 85 c0 75 46 48 85 db > > > 74 41 > > > 49 8b 34 24 <48> 8b 96 a0 00 00 00 0f b7 52 08 66 c1 c2 08 66 81 fa > > > be 3e 74 > > > [Tue Jun 13 13:18:48 2017] RIP [<ffffffffc06555a0>] > > > xfs_da3_node_read+0x30/0xb0 [xfs] > > > [Tue Jun 13 13:18:48 2017] RSP <ffff883fbda0bc88> > > > [Tue Jun 13 13:18:48 2017] CR2: 00000000000000a0 > > > [Tue Jun 13 13:18:48 2017] ---[ end trace 5470d0d55cacb4ef ]--- > > > > > > The ceph OSD running on this server has then the issue that it can > > > not reach > > > any other osd in the pool. > > > > > > -1043> 2017-06-13 13:24:00.917597 7fc539a72700 0 -- > > > 192.168.14.19:6827/3389 >> 192.168.14.7:6805/3658 > > > pipe(0x558219846000 sd=23 > > > :6827 > > > s=0 pgs=0 cs=0 l=0 c=0x55821a330400).accept connect_seq 7 vs > > > existing 7 > > > state standby > > > -1042> 2017-06-13 13:24:00.918433 7fc539a72700 0 -- > > > 192.168.14.19:6827/3389 >> 192.168.14.7:6805/3658 > > > pipe(0x558219846000 sd=23 > > > :6827 > > > s=0 pgs=0 cs=0 l=0 c=0x55821a330400).accept connect_seq 8 vs > > > existing 7 > > > state standby > > > -1041> 2017-06-13 13:24:03.654983 7fc4dd21d700 0 -- > > > 192.168.14.19:6825/3389 >> :/0 pipe(0x5581fa6ba000 sd=524 :6825 s=0 > > > pgs=0 > > > cs=0 l=0 > > > c=0x55820a9e5000).accept failed to getpeername (107) Transport > > > endpoint is > > > not connected > > > > > > > > > There are a lot more of these messages. Does any of you have the > > > same issue? > > > We are running Ubuntu 16.04 with kernel 4.4.0-75.96. > > > > > > Best regards, > > > Jonas > > > -- > > > To unsubscribe from this list: send the line "unsubscribe linux-xfs" > > > in > > > the body of a message to majordomo@vger.kernel.org > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: XFS Calltraces by using XFS with Ceph 2017-06-14 14:07 ` Brian Foster @ 2017-06-14 15:55 ` Darrick J. Wong 2017-06-15 5:55 ` list 0 siblings, 1 reply; 12+ messages in thread From: Darrick J. Wong @ 2017-06-14 15:55 UTC (permalink / raw) To: Brian Foster; +Cc: list, linux-xfs On Wed, Jun 14, 2017 at 10:07:32AM -0400, Brian Foster wrote: > On Wed, Jun 14, 2017 at 03:22:36PM +0200, list@jonas-server.de wrote: > > I get this output of gdb: > > > > # gdb /usr/lib/debug/lib/modules/4.4.0-75-generic/kernel/fs/xfs/xfs.ko > > GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1 > > [...] > > Reading symbols from > > /usr/lib/debug/lib/modules/4.4.0-75-generic/kernel/fs/xfs/xfs.ko...done. > > (gdb) list *xfs_da3_node_read+0x30 > > 0x2b5d0 is in xfs_da3_node_read > > (/build/linux-Hlembm/linux-4.4.0/fs/xfs/libxfs/xfs_da_btree.c:270). > > 265 /build/linux-Hlembm/linux-4.4.0/fs/xfs/libxfs/xfs_da_btree.c: No such > > file or directory. > > > > This would be more helpful if you had the source code available. :P > > Anyways, I suspect you have a NULL buffer (though I'm not sure where the > 0xa0 offset comes from). There have been a couple fixes in that area > that come to mind, but it looks to me that v4.4 kernels should already > have them. Otherwise, this doesn't ring any bells for me. Perhaps > somebody else can chime in on that. > > I suppose the best next step is to try a more recent, non-distro kernel. > If the problem still occurs, see if you can provide a crash dump for > analysis. I wonder if this is a longstanding quirk of the dabtree reader routines where they call xfs_trans_buf_set_type() after xfs_da_read_buf() without actually checking that *bpp point to a buffer, which is what you get if the fork offset maps to a hole. In theory the dabtree shouldn't ever point to a hole, but I've seen occasional bug reports about that happening, and we could do better than just crashing. :) (I was working on a patch to fix all the places where we stumble over a NULL bp, but it produced xfstest regressions and then I got distracted.) Looking at the new Elixir[1], it looks like we're trying to deref ((*bpp)->b_addr)->magic, so that might explain the crash you see. --D [1] http://elixir.free-electrons.com/linux/v4.4.72/source/fs/xfs/libxfs/xfs_da_btree.c#L270 > > Brian > > > Am 2017-06-14 14:08, schrieb Brian Foster: > > > On Wed, Jun 14, 2017 at 10:22:38AM +0200, list@jonas-server.de wrote: > > > > Hello guys, > > > > > > > > we have currently an issue with our ceph setup based on XFS. > > > > Sometimes some > > > > nodes are dying with high load with this calltrace in dmesg: > > > > > > > > [Tue Jun 13 13:18:48 2017] BUG: unable to handle kernel NULL pointer > > > > dereference at 00000000000000a0 > > > > [Tue Jun 13 13:18:48 2017] IP: [<ffffffffc06555a0>] > > > > xfs_da3_node_read+0x30/0xb0 [xfs] > > > > [Tue Jun 13 13:18:48 2017] PGD 0 > > > > [Tue Jun 13 13:18:48 2017] Oops: 0000 [#1] SMP > > > > [Tue Jun 13 13:18:48 2017] Modules linked in: cpuid arc4 md4 > > > > nls_utf8 cifs > > > > fscache nfnetlink_queue nfnetlink xt_CHECKSUM xt_nat iptable_nat > > > > nf_nat_ipv4 > > > > xt_NFQUEUE xt_CLASSIFY ip6table_mangle dccp_diag dccp tcp_diag > > > > udp_diag > > > > inet_diag unix_diag af_packet_diag netlink_diag veth dummy bridge > > > > stp llc > > > > ebtable_filter ebtables iptable_mangle xt_CT iptable_raw > > > > nf_conntrack_ipv4 > > > > nf_defrag_ipv4 iptable_filter ip_tables xt_tcpudp nf_conntrack_ipv6 > > > > nf_defrag_ipv6 xt_conntrack ip6table_filter ip6_tables x_tables xfs > > > > ipmi_devintf dcdbas x86_pkg_temp_thermal intel_powerclamp coretemp > > > > crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel > > > > ipmi_ssif > > > > aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac > > > > edac_core > > > > input_leds joydev lpc_ich ioatdma shpchp 8250_fintek ipmi_si > > > > ipmi_msghandler > > > > acpi_pad acpi_power_meter > > > > [Tue Jun 13 13:18:48 2017] mac_hid vhost_net vhost macvtap macvlan > > > > kvm_intel kvm irqbypass cdc_ether nf_nat_ftp tcp_htcp nf_nat_pptp > > > > nf_nat_proto_gre nf_conntrack_ftp bonding nf_nat_sip > > > > nf_conntrack_sip nf_nat > > > > nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack usbnet mii lp > > > > parport > > > > autofs4 btrfs raid456 async_raid6_recov async_memcpy async_pq > > > > async_xor > > > > async_tx xor raid6_pq libcrc32c raid0 multipath linear raid10 raid1 > > > > hid_generic usbhid hid ixgbe igb vxlan ip6_udp_tunnel ahci dca > > > > udp_tunnel > > > > libahci i2c_algo_bit ptp megaraid_sas pps_core mdio wmi fjes > > > > [Tue Jun 13 13:18:48 2017] CPU: 3 PID: 3844 Comm: tp_fstore_op Not > > > > tainted > > > > 4.4.0-75-generic #96-Ubuntu > > > > [Tue Jun 13 13:18:48 2017] Hardware name: Dell Inc. PowerEdge > > > > R720/0XH7F2, > > > > BIOS 2.5.4 01/22/2016 > > > > [Tue Jun 13 13:18:48 2017] task: ffff881feda65400 ti: ffff883fbda08000 > > > > task.ti: ffff883fbda08000 > > > > [Tue Jun 13 13:18:48 2017] RIP: 0010:[<ffffffffc06555a0>] > > > > [<ffffffffc06555a0>] xfs_da3_node_read+0x30/0xb0 [xfs] > > > > > > What line does this point at (i.e., 'list *xfs_da3_node_read+0x30' from > > > gdb) on your kernel? > > > > > > Brian > > > > > > > [Tue Jun 13 13:18:48 2017] RSP: 0018:ffff883fbda0bc88 EFLAGS: > > > > 00010286 > > > > [Tue Jun 13 13:18:48 2017] RAX: 0000000000000000 RBX: > > > > ffff8801102c5050 RCX: > > > > 0000000000000001 > > > > [Tue Jun 13 13:18:48 2017] RDX: 0000000000000000 RSI: > > > > 0000000000000000 RDI: > > > > ffff883fbda0bc38 > > > > [Tue Jun 13 13:18:48 2017] RBP: ffff883fbda0bca8 R08: > > > > 0000000000000001 R09: > > > > fffffffffffffffe > > > > [Tue Jun 13 13:18:48 2017] R10: ffff880007374ae0 R11: > > > > 0000000000000001 R12: > > > > ffff883fbda0bcd8 > > > > [Tue Jun 13 13:18:48 2017] R13: ffff880035ac4c80 R14: > > > > 0000000000000001 R15: > > > > 000000008b1f4885 > > > > [Tue Jun 13 13:18:48 2017] FS: 00007fc574607700(0000) > > > > GS:ffff883fff040000(0000) knlGS:0000000000000000 > > > > [Tue Jun 13 13:18:48 2017] CS: 0010 DS: 0000 ES: 0000 CR0: > > > > 0000000080050033 > > > > [Tue Jun 13 13:18:48 2017] CR2: 00000000000000a0 CR3: > > > > 0000003fd828d000 CR4: > > > > 00000000001426e0 > > > > [Tue Jun 13 13:18:48 2017] Stack: > > > > [Tue Jun 13 13:18:48 2017] ffffffffc06b4b50 ffffffffc0695ecc > > > > ffff883fbda0bde0 0000000000000001 > > > > [Tue Jun 13 13:18:48 2017] ffff883fbda0bd20 ffffffffc06718b3 > > > > 0000000300000008 ffff880e99b44010 > > > > [Tue Jun 13 13:18:48 2017] 00000000360c65a8 ffff88270f80b900 > > > > 0000000000000000 0000000000000000 > > > > [Tue Jun 13 13:18:48 2017] Call Trace: > > > > [Tue Jun 13 13:18:48 2017] [<ffffffffc0695ecc>] ? > > > > xfs_trans_roll+0x2c/0x50 > > > > [xfs] > > > > [Tue Jun 13 13:18:48 2017] [<ffffffffc06718b3>] > > > > xfs_attr3_node_inactive+0x183/0x220 [xfs] > > > > [Tue Jun 13 13:18:48 2017] [<ffffffffc06718f9>] > > > > xfs_attr3_node_inactive+0x1c9/0x220 [xfs] > > > > [Tue Jun 13 13:18:48 2017] [<ffffffffc06719fc>] > > > > xfs_attr3_root_inactive+0xac/0x100 [xfs] > > > > [Tue Jun 13 13:18:48 2017] [<ffffffffc0671b9c>] > > > > xfs_attr_inactive+0x14c/0x1a0 [xfs] > > > > [Tue Jun 13 13:18:48 2017] [<ffffffffc068bda5>] > > > > xfs_inactive+0x85/0x120 > > > > [xfs] > > > > [Tue Jun 13 13:18:48 2017] [<ffffffffc06912f5>] > > > > xfs_fs_evict_inode+0xa5/0x100 [xfs] > > > > [Tue Jun 13 13:18:48 2017] [<ffffffff8122a90e>] evict+0xbe/0x190 > > > > [Tue Jun 13 13:18:48 2017] [<ffffffff8122abf1>] iput+0x1c1/0x240 > > > > [Tue Jun 13 13:18:48 2017] [<ffffffff8121f6b9>] > > > > do_unlinkat+0x199/0x2d0 > > > > [Tue Jun 13 13:18:48 2017] [<ffffffff81220256>] SyS_unlink+0x16/0x20 > > > > [Tue Jun 13 13:18:48 2017] [<ffffffff8183b972>] > > > > entry_SYSCALL_64_fastpath+0x16/0x71 > > > > [Tue Jun 13 13:18:48 2017] Code: 55 48 89 e5 41 54 53 4d 89 c4 48 89 > > > > fb 48 > > > > 83 ec 10 48 c7 04 24 50 4b 6b c0 e8 dd fe ff ff 85 c0 75 46 48 85 db > > > > 74 41 > > > > 49 8b 34 24 <48> 8b 96 a0 00 00 00 0f b7 52 08 66 c1 c2 08 66 81 fa > > > > be 3e 74 > > > > [Tue Jun 13 13:18:48 2017] RIP [<ffffffffc06555a0>] > > > > xfs_da3_node_read+0x30/0xb0 [xfs] > > > > [Tue Jun 13 13:18:48 2017] RSP <ffff883fbda0bc88> > > > > [Tue Jun 13 13:18:48 2017] CR2: 00000000000000a0 > > > > [Tue Jun 13 13:18:48 2017] ---[ end trace 5470d0d55cacb4ef ]--- > > > > > > > > The ceph OSD running on this server has then the issue that it can > > > > not reach > > > > any other osd in the pool. > > > > > > > > -1043> 2017-06-13 13:24:00.917597 7fc539a72700 0 -- > > > > 192.168.14.19:6827/3389 >> 192.168.14.7:6805/3658 > > > > pipe(0x558219846000 sd=23 > > > > :6827 > > > > s=0 pgs=0 cs=0 l=0 c=0x55821a330400).accept connect_seq 7 vs > > > > existing 7 > > > > state standby > > > > -1042> 2017-06-13 13:24:00.918433 7fc539a72700 0 -- > > > > 192.168.14.19:6827/3389 >> 192.168.14.7:6805/3658 > > > > pipe(0x558219846000 sd=23 > > > > :6827 > > > > s=0 pgs=0 cs=0 l=0 c=0x55821a330400).accept connect_seq 8 vs > > > > existing 7 > > > > state standby > > > > -1041> 2017-06-13 13:24:03.654983 7fc4dd21d700 0 -- > > > > 192.168.14.19:6825/3389 >> :/0 pipe(0x5581fa6ba000 sd=524 :6825 s=0 > > > > pgs=0 > > > > cs=0 l=0 > > > > c=0x55820a9e5000).accept failed to getpeername (107) Transport > > > > endpoint is > > > > not connected > > > > > > > > > > > > There are a lot more of these messages. Does any of you have the > > > > same issue? > > > > We are running Ubuntu 16.04 with kernel 4.4.0-75.96. > > > > > > > > Best regards, > > > > Jonas > > > > -- > > > > To unsubscribe from this list: send the line "unsubscribe linux-xfs" > > > > in > > > > the body of a message to majordomo@vger.kernel.org > > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: XFS Calltraces by using XFS with Ceph 2017-06-14 15:55 ` Darrick J. Wong @ 2017-06-15 5:55 ` list 2017-06-15 17:20 ` Emmanuel Florac 0 siblings, 1 reply; 12+ messages in thread From: list @ 2017-06-15 5:55 UTC (permalink / raw) To: Darrick J. Wong; +Cc: Brian Foster, linux-xfs, linux-xfs-owner Am 2017-06-14 17:55, schrieb Darrick J. Wong: > On Wed, Jun 14, 2017 at 10:07:32AM -0400, Brian Foster wrote: >> On Wed, Jun 14, 2017 at 03:22:36PM +0200, list@jonas-server.de wrote: >> > I get this output of gdb: >> > >> > # gdb /usr/lib/debug/lib/modules/4.4.0-75-generic/kernel/fs/xfs/xfs.ko >> > GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1 >> > [...] >> > Reading symbols from >> > /usr/lib/debug/lib/modules/4.4.0-75-generic/kernel/fs/xfs/xfs.ko...done. >> > (gdb) list *xfs_da3_node_read+0x30 >> > 0x2b5d0 is in xfs_da3_node_read >> > (/build/linux-Hlembm/linux-4.4.0/fs/xfs/libxfs/xfs_da_btree.c:270). >> > 265 /build/linux-Hlembm/linux-4.4.0/fs/xfs/libxfs/xfs_da_btree.c: No such >> > file or directory. >> > >> >> This would be more helpful if you had the source code available. :P I've figured out on how to get the source code listed :) (gdb) list *xfs_da3_node_read+0x30 0x2b5d0 is in xfs_da3_node_read (/build/linux-Hlembm/linux-4.4.0/fs/xfs/libxfs/xfs_da_btree.c:270). 265 which_fork, &xfs_da3_node_buf_ops); 266 if (!err && tp) { 267 struct xfs_da_blkinfo *info = (*bpp)->b_addr; 268 int type; 269 270 switch (be16_to_cpu(info->magic)) { 271 case XFS_DA_NODE_MAGIC: 272 case XFS_DA3_NODE_MAGIC: 273 type = XFS_BLFT_DA_NODE_BUF; 274 break; Maybe this helps to investigate the calltrace further. >> >> Anyways, I suspect you have a NULL buffer (though I'm not sure where >> the >> 0xa0 offset comes from). There have been a couple fixes in that area >> that come to mind, but it looks to me that v4.4 kernels should already >> have them. Otherwise, this doesn't ring any bells for me. Perhaps >> somebody else can chime in on that. >> >> I suppose the best next step is to try a more recent, non-distro >> kernel. >> If the problem still occurs, see if you can provide a crash dump for >> analysis. > > I wonder if this is a longstanding quirk of the dabtree reader routines > where they call xfs_trans_buf_set_type() after xfs_da_read_buf() > without > actually checking that *bpp point to a buffer, which is what you get if > the fork offset maps to a hole. In theory the dabtree shouldn't ever > point to a hole, but I've seen occasional bug reports about that > happening, and we could do better than just crashing. :) > > (I was working on a patch to fix all the places where we stumble over a > NULL bp, but it produced xfstest regressions and then I got > distracted.) > > Looking at the new Elixir[1], it looks like we're trying to deref > ((*bpp)->b_addr)->magic, so that might explain the crash you see. > > --D > > [1] > http://elixir.free-electrons.com/linux/v4.4.72/source/fs/xfs/libxfs/xfs_da_btree.c#L270 > >> >> Brian >> >> > Am 2017-06-14 14:08, schrieb Brian Foster: >> > > On Wed, Jun 14, 2017 at 10:22:38AM +0200, list@jonas-server.de wrote: >> > > > Hello guys, >> > > > >> > > > we have currently an issue with our ceph setup based on XFS. >> > > > Sometimes some >> > > > nodes are dying with high load with this calltrace in dmesg: >> > > > >> > > > [Tue Jun 13 13:18:48 2017] BUG: unable to handle kernel NULL pointer >> > > > dereference at 00000000000000a0 >> > > > [Tue Jun 13 13:18:48 2017] IP: [<ffffffffc06555a0>] >> > > > xfs_da3_node_read+0x30/0xb0 [xfs] >> > > > [Tue Jun 13 13:18:48 2017] PGD 0 >> > > > [Tue Jun 13 13:18:48 2017] Oops: 0000 [#1] SMP >> > > > [Tue Jun 13 13:18:48 2017] Modules linked in: cpuid arc4 md4 >> > > > nls_utf8 cifs >> > > > fscache nfnetlink_queue nfnetlink xt_CHECKSUM xt_nat iptable_nat >> > > > nf_nat_ipv4 >> > > > xt_NFQUEUE xt_CLASSIFY ip6table_mangle dccp_diag dccp tcp_diag >> > > > udp_diag >> > > > inet_diag unix_diag af_packet_diag netlink_diag veth dummy bridge >> > > > stp llc >> > > > ebtable_filter ebtables iptable_mangle xt_CT iptable_raw >> > > > nf_conntrack_ipv4 >> > > > nf_defrag_ipv4 iptable_filter ip_tables xt_tcpudp nf_conntrack_ipv6 >> > > > nf_defrag_ipv6 xt_conntrack ip6table_filter ip6_tables x_tables xfs >> > > > ipmi_devintf dcdbas x86_pkg_temp_thermal intel_powerclamp coretemp >> > > > crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel >> > > > ipmi_ssif >> > > > aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac >> > > > edac_core >> > > > input_leds joydev lpc_ich ioatdma shpchp 8250_fintek ipmi_si >> > > > ipmi_msghandler >> > > > acpi_pad acpi_power_meter >> > > > [Tue Jun 13 13:18:48 2017] mac_hid vhost_net vhost macvtap macvlan >> > > > kvm_intel kvm irqbypass cdc_ether nf_nat_ftp tcp_htcp nf_nat_pptp >> > > > nf_nat_proto_gre nf_conntrack_ftp bonding nf_nat_sip >> > > > nf_conntrack_sip nf_nat >> > > > nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack usbnet mii lp >> > > > parport >> > > > autofs4 btrfs raid456 async_raid6_recov async_memcpy async_pq >> > > > async_xor >> > > > async_tx xor raid6_pq libcrc32c raid0 multipath linear raid10 raid1 >> > > > hid_generic usbhid hid ixgbe igb vxlan ip6_udp_tunnel ahci dca >> > > > udp_tunnel >> > > > libahci i2c_algo_bit ptp megaraid_sas pps_core mdio wmi fjes >> > > > [Tue Jun 13 13:18:48 2017] CPU: 3 PID: 3844 Comm: tp_fstore_op Not >> > > > tainted >> > > > 4.4.0-75-generic #96-Ubuntu >> > > > [Tue Jun 13 13:18:48 2017] Hardware name: Dell Inc. PowerEdge >> > > > R720/0XH7F2, >> > > > BIOS 2.5.4 01/22/2016 >> > > > [Tue Jun 13 13:18:48 2017] task: ffff881feda65400 ti: ffff883fbda08000 >> > > > task.ti: ffff883fbda08000 >> > > > [Tue Jun 13 13:18:48 2017] RIP: 0010:[<ffffffffc06555a0>] >> > > > [<ffffffffc06555a0>] xfs_da3_node_read+0x30/0xb0 [xfs] >> > > >> > > What line does this point at (i.e., 'list *xfs_da3_node_read+0x30' from >> > > gdb) on your kernel? >> > > >> > > Brian >> > > >> > > > [Tue Jun 13 13:18:48 2017] RSP: 0018:ffff883fbda0bc88 EFLAGS: >> > > > 00010286 >> > > > [Tue Jun 13 13:18:48 2017] RAX: 0000000000000000 RBX: >> > > > ffff8801102c5050 RCX: >> > > > 0000000000000001 >> > > > [Tue Jun 13 13:18:48 2017] RDX: 0000000000000000 RSI: >> > > > 0000000000000000 RDI: >> > > > ffff883fbda0bc38 >> > > > [Tue Jun 13 13:18:48 2017] RBP: ffff883fbda0bca8 R08: >> > > > 0000000000000001 R09: >> > > > fffffffffffffffe >> > > > [Tue Jun 13 13:18:48 2017] R10: ffff880007374ae0 R11: >> > > > 0000000000000001 R12: >> > > > ffff883fbda0bcd8 >> > > > [Tue Jun 13 13:18:48 2017] R13: ffff880035ac4c80 R14: >> > > > 0000000000000001 R15: >> > > > 000000008b1f4885 >> > > > [Tue Jun 13 13:18:48 2017] FS: 00007fc574607700(0000) >> > > > GS:ffff883fff040000(0000) knlGS:0000000000000000 >> > > > [Tue Jun 13 13:18:48 2017] CS: 0010 DS: 0000 ES: 0000 CR0: >> > > > 0000000080050033 >> > > > [Tue Jun 13 13:18:48 2017] CR2: 00000000000000a0 CR3: >> > > > 0000003fd828d000 CR4: >> > > > 00000000001426e0 >> > > > [Tue Jun 13 13:18:48 2017] Stack: >> > > > [Tue Jun 13 13:18:48 2017] ffffffffc06b4b50 ffffffffc0695ecc >> > > > ffff883fbda0bde0 0000000000000001 >> > > > [Tue Jun 13 13:18:48 2017] ffff883fbda0bd20 ffffffffc06718b3 >> > > > 0000000300000008 ffff880e99b44010 >> > > > [Tue Jun 13 13:18:48 2017] 00000000360c65a8 ffff88270f80b900 >> > > > 0000000000000000 0000000000000000 >> > > > [Tue Jun 13 13:18:48 2017] Call Trace: >> > > > [Tue Jun 13 13:18:48 2017] [<ffffffffc0695ecc>] ? >> > > > xfs_trans_roll+0x2c/0x50 >> > > > [xfs] >> > > > [Tue Jun 13 13:18:48 2017] [<ffffffffc06718b3>] >> > > > xfs_attr3_node_inactive+0x183/0x220 [xfs] >> > > > [Tue Jun 13 13:18:48 2017] [<ffffffffc06718f9>] >> > > > xfs_attr3_node_inactive+0x1c9/0x220 [xfs] >> > > > [Tue Jun 13 13:18:48 2017] [<ffffffffc06719fc>] >> > > > xfs_attr3_root_inactive+0xac/0x100 [xfs] >> > > > [Tue Jun 13 13:18:48 2017] [<ffffffffc0671b9c>] >> > > > xfs_attr_inactive+0x14c/0x1a0 [xfs] >> > > > [Tue Jun 13 13:18:48 2017] [<ffffffffc068bda5>] >> > > > xfs_inactive+0x85/0x120 >> > > > [xfs] >> > > > [Tue Jun 13 13:18:48 2017] [<ffffffffc06912f5>] >> > > > xfs_fs_evict_inode+0xa5/0x100 [xfs] >> > > > [Tue Jun 13 13:18:48 2017] [<ffffffff8122a90e>] evict+0xbe/0x190 >> > > > [Tue Jun 13 13:18:48 2017] [<ffffffff8122abf1>] iput+0x1c1/0x240 >> > > > [Tue Jun 13 13:18:48 2017] [<ffffffff8121f6b9>] >> > > > do_unlinkat+0x199/0x2d0 >> > > > [Tue Jun 13 13:18:48 2017] [<ffffffff81220256>] SyS_unlink+0x16/0x20 >> > > > [Tue Jun 13 13:18:48 2017] [<ffffffff8183b972>] >> > > > entry_SYSCALL_64_fastpath+0x16/0x71 >> > > > [Tue Jun 13 13:18:48 2017] Code: 55 48 89 e5 41 54 53 4d 89 c4 48 89 >> > > > fb 48 >> > > > 83 ec 10 48 c7 04 24 50 4b 6b c0 e8 dd fe ff ff 85 c0 75 46 48 85 db >> > > > 74 41 >> > > > 49 8b 34 24 <48> 8b 96 a0 00 00 00 0f b7 52 08 66 c1 c2 08 66 81 fa >> > > > be 3e 74 >> > > > [Tue Jun 13 13:18:48 2017] RIP [<ffffffffc06555a0>] >> > > > xfs_da3_node_read+0x30/0xb0 [xfs] >> > > > [Tue Jun 13 13:18:48 2017] RSP <ffff883fbda0bc88> >> > > > [Tue Jun 13 13:18:48 2017] CR2: 00000000000000a0 >> > > > [Tue Jun 13 13:18:48 2017] ---[ end trace 5470d0d55cacb4ef ]--- >> > > > >> > > > The ceph OSD running on this server has then the issue that it can >> > > > not reach >> > > > any other osd in the pool. >> > > > >> > > > -1043> 2017-06-13 13:24:00.917597 7fc539a72700 0 -- >> > > > 192.168.14.19:6827/3389 >> 192.168.14.7:6805/3658 >> > > > pipe(0x558219846000 sd=23 >> > > > :6827 >> > > > s=0 pgs=0 cs=0 l=0 c=0x55821a330400).accept connect_seq 7 vs >> > > > existing 7 >> > > > state standby >> > > > -1042> 2017-06-13 13:24:00.918433 7fc539a72700 0 -- >> > > > 192.168.14.19:6827/3389 >> 192.168.14.7:6805/3658 >> > > > pipe(0x558219846000 sd=23 >> > > > :6827 >> > > > s=0 pgs=0 cs=0 l=0 c=0x55821a330400).accept connect_seq 8 vs >> > > > existing 7 >> > > > state standby >> > > > -1041> 2017-06-13 13:24:03.654983 7fc4dd21d700 0 -- >> > > > 192.168.14.19:6825/3389 >> :/0 pipe(0x5581fa6ba000 sd=524 :6825 s=0 >> > > > pgs=0 >> > > > cs=0 l=0 >> > > > c=0x55820a9e5000).accept failed to getpeername (107) Transport >> > > > endpoint is >> > > > not connected >> > > > >> > > > >> > > > There are a lot more of these messages. Does any of you have the >> > > > same issue? >> > > > We are running Ubuntu 16.04 with kernel 4.4.0-75.96. >> > > > >> > > > Best regards, >> > > > Jonas >> > > > -- >> > > > To unsubscribe from this list: send the line "unsubscribe linux-xfs" >> > > > in >> > > > the body of a message to majordomo@vger.kernel.org >> > > > More majordomo info at http://vger.kernel.org/majordomo-info.html >> > -- >> > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in >> > the body of a message to majordomo@vger.kernel.org >> > More majordomo info at http://vger.kernel.org/majordomo-info.html >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" >> in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: XFS Calltraces by using XFS with Ceph 2017-06-15 5:55 ` list @ 2017-06-15 17:20 ` Emmanuel Florac 2017-06-16 5:37 ` list 0 siblings, 1 reply; 12+ messages in thread From: Emmanuel Florac @ 2017-06-15 17:20 UTC (permalink / raw) To: list; +Cc: Darrick J. Wong, Brian Foster, linux-xfs, linux-xfs-owner [-- Attachment #1: Type: text/plain, Size: 844 bytes --] Le Thu, 15 Jun 2017 07:55:12 +0200 list@jonas-server.de écrivait: > > Looking at the new Elixir[1], it looks like we're trying to deref > > ((*bpp)->b_addr)->magic, so that might explain the crash you see. > > > > --D > > > > [1] > > http://elixir.free-electrons.com/linux/v4.4.72/source/fs/xfs/libxfs/xfs_da_btree.c#L270 > > > >> Indeed that matches exactly the crash Darrick mentioned. Looks like you need either a special hand-patched kernel... or wait for an official fix for this one :/ -- ------------------------------------------------------------------------ Emmanuel Florac | Direction technique | Intellique | <eflorac@intellique.com> | +33 1 78 94 84 02 ------------------------------------------------------------------------ [-- Attachment #2: Signature digitale OpenPGP --] [-- Type: application/pgp-signature, Size: 181 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: XFS Calltraces by using XFS with Ceph 2017-06-15 17:20 ` Emmanuel Florac @ 2017-06-16 5:37 ` list 2017-06-16 13:15 ` Emmanuel Florac 2017-06-16 17:46 ` Darrick J. Wong 0 siblings, 2 replies; 12+ messages in thread From: list @ 2017-06-16 5:37 UTC (permalink / raw) To: Emmanuel Florac; +Cc: Darrick J. Wong, Brian Foster, linux-xfs, linux-xfs-owner Am 2017-06-15 19:20, schrieb Emmanuel Florac: > Le Thu, 15 Jun 2017 07:55:12 +0200 > list@jonas-server.de écrivait: > >> > Looking at the new Elixir[1], it looks like we're trying to deref >> > ((*bpp)->b_addr)->magic, so that might explain the crash you see. >> > >> > --D >> > >> > [1] >> > http://elixir.free-electrons.com/linux/v4.4.72/source/fs/xfs/libxfs/xfs_da_btree.c#L270 >> > >> >> > > Indeed that matches exactly the crash Darrick mentioned. Looks like you > need either a special hand-patched kernel... or wait for an official > fix for this one :/ When will a fix normally be in the ubuntu kernel? I need to have a stable distribution kernel, I can not install a patched version. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: XFS Calltraces by using XFS with Ceph 2017-06-16 5:37 ` list @ 2017-06-16 13:15 ` Emmanuel Florac 2017-06-16 17:46 ` Darrick J. Wong 1 sibling, 0 replies; 12+ messages in thread From: Emmanuel Florac @ 2017-06-16 13:15 UTC (permalink / raw) To: list; +Cc: Darrick J. Wong, Brian Foster, linux-xfs, linux-xfs-owner [-- Attachment #1: Type: text/plain, Size: 1199 bytes --] Le Fri, 16 Jun 2017 07:37:11 +0200 list@jonas-server.de écrivait: > > > > Indeed that matches exactly the crash Darrick mentioned. Looks like > > you need either a special hand-patched kernel... or wait for an > > official fix for this one :/ > > When will a fix normally be in the ubuntu kernel? I need to have a > stable distribution kernel, I can not install a patched version. Difficult to know. From the moment an update is released, I suppose it makes it to Ubuntu updates in a matter of a few days. But first, we'd need a patch that could be submitted to mainline. the patch needs to be tested under conditions that may trigger the problem (you seem to be the perfect guinea pig, here), then deemed a "stable" correction to be applied to all maintained kernels, then backported to 4.4. The complete process could be quite long indeed. -- ------------------------------------------------------------------------ Emmanuel Florac | Direction technique | Intellique | <eflorac@intellique.com> | +33 1 78 94 84 02 ------------------------------------------------------------------------ [-- Attachment #2: Signature digitale OpenPGP --] [-- Type: application/pgp-signature, Size: 181 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: XFS Calltraces by using XFS with Ceph 2017-06-16 5:37 ` list 2017-06-16 13:15 ` Emmanuel Florac @ 2017-06-16 17:46 ` Darrick J. Wong 2017-06-19 10:17 ` list 1 sibling, 1 reply; 12+ messages in thread From: Darrick J. Wong @ 2017-06-16 17:46 UTC (permalink / raw) To: list; +Cc: Emmanuel Florac, Brian Foster, linux-xfs, linux-xfs-owner On Fri, Jun 16, 2017 at 07:37:11AM +0200, list@jonas-server.de wrote: > Am 2017-06-15 19:20, schrieb Emmanuel Florac: > >Le Thu, 15 Jun 2017 07:55:12 +0200 > >list@jonas-server.de écrivait: > > > >>> Looking at the new Elixir[1], it looks like we're trying to deref > >>> ((*bpp)->b_addr)->magic, so that might explain the crash you see. > >>> > >>> --D > >>> > >>> [1] > >>> http://elixir.free-electrons.com/linux/v4.4.72/source/fs/xfs/libxfs/xfs_da_btree.c#L270 > >>> > >>>> > > > >Indeed that matches exactly the crash Darrick mentioned. Looks like you > >need either a special hand-patched kernel... or wait for an official > >fix for this one :/ > > When will a fix normally be in the ubuntu kernel? I need to have a stable > distribution kernel, I can not install a patched version. Like Emmanuel said, first a fix has to go upstream, then it can go to the 4.4 stable kernel, and then it'll probably end up in the Ubuntu kernel. Can you post a metadump of the xfs filesystem somewhere for further analysis? I'm pretty sure it's the same problem reported by David Shaw on this list a couple of weeks ago. Since you can't run one-off test kernels, send us a metadump for recreation in a lab is the only other way to figure out if the fix works. (And yes, I /do/ have a lightly tested fix which I'm about to send out.) --D > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: XFS Calltraces by using XFS with Ceph 2017-06-16 17:46 ` Darrick J. Wong @ 2017-06-19 10:17 ` list 2017-07-10 6:34 ` list 0 siblings, 1 reply; 12+ messages in thread From: list @ 2017-06-19 10:17 UTC (permalink / raw) To: Darrick J. Wong; +Cc: Emmanuel Florac, Brian Foster, linux-xfs, linux-xfs-owner Am 2017-06-16 19:46, schrieb Darrick J. Wong: > On Fri, Jun 16, 2017 at 07:37:11AM +0200, list@jonas-server.de wrote: >> Am 2017-06-15 19:20, schrieb Emmanuel Florac: >> >Le Thu, 15 Jun 2017 07:55:12 +0200 >> >list@jonas-server.de écrivait: >> > >> >>> Looking at the new Elixir[1], it looks like we're trying to deref >> >>> ((*bpp)->b_addr)->magic, so that might explain the crash you see. >> >>> >> >>> --D >> >>> >> >>> [1] >> >>> http://elixir.free-electrons.com/linux/v4.4.72/source/fs/xfs/libxfs/xfs_da_btree.c#L270 >> >>> >> >>>> >> > >> >Indeed that matches exactly the crash Darrick mentioned. Looks like you >> >need either a special hand-patched kernel... or wait for an official >> >fix for this one :/ >> >> When will a fix normally be in the ubuntu kernel? I need to have a >> stable >> distribution kernel, I can not install a patched version. > > Like Emmanuel said, first a fix has to go upstream, then it can go to > the > 4.4 stable kernel, and then it'll probably end up in the Ubuntu kernel. > > Can you post a metadump of the xfs filesystem somewhere for further > analysis? I'm pretty sure it's the same problem reported by David Shaw > on this list a couple of weeks ago. Since you can't run one-off test > kernels, send us a metadump for recreation in a lab is the only other > way to figure out if the fix works. So it is hard for me to run tests with untested kernels as the system is already used in production. I've created a metadump from the xfs filesystem and uploaded it: https://transfer.sh/XdlZ5/xfs_metadump.gz I hope it helps you trigger the bug during your tests. > > (And yes, I /do/ have a lightly tested fix which I'm about to send > out.) > > --D > >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" >> in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: XFS Calltraces by using XFS with Ceph 2017-06-19 10:17 ` list @ 2017-07-10 6:34 ` list 0 siblings, 0 replies; 12+ messages in thread From: list @ 2017-07-10 6:34 UTC (permalink / raw) To: Darrick J. Wong; +Cc: Emmanuel Florac, Brian Foster, linux-xfs, linux-xfs-owner Am 2017-06-19 12:17, schrieb list@jonas-server.de: > Am 2017-06-16 19:46, schrieb Darrick J. Wong: >> On Fri, Jun 16, 2017 at 07:37:11AM +0200, list@jonas-server.de wrote: >>> Am 2017-06-15 19:20, schrieb Emmanuel Florac: >>> >Le Thu, 15 Jun 2017 07:55:12 +0200 >>> >list@jonas-server.de écrivait: >>> > >>> >>> Looking at the new Elixir[1], it looks like we're trying to deref >>> >>> ((*bpp)->b_addr)->magic, so that might explain the crash you see. >>> >>> >>> >>> --D >>> >>> >>> >>> [1] >>> >>> http://elixir.free-electrons.com/linux/v4.4.72/source/fs/xfs/libxfs/xfs_da_btree.c#L270 >>> >>> >>> >>>> >>> > >>> >Indeed that matches exactly the crash Darrick mentioned. Looks like you >>> >need either a special hand-patched kernel... or wait for an official >>> >fix for this one :/ >>> >>> When will a fix normally be in the ubuntu kernel? I need to have a >>> stable >>> distribution kernel, I can not install a patched version. >> >> Like Emmanuel said, first a fix has to go upstream, then it can go to >> the >> 4.4 stable kernel, and then it'll probably end up in the Ubuntu >> kernel. >> >> Can you post a metadump of the xfs filesystem somewhere for further >> analysis? I'm pretty sure it's the same problem reported by David >> Shaw >> on this list a couple of weeks ago. Since you can't run one-off test >> kernels, send us a metadump for recreation in a lab is the only other >> way to figure out if the fix works. > > So it is hard for me to run tests with untested kernels as the system > is already > used in production. I've created a metadump from the xfs filesystem and > uploaded it: > https://transfer.sh/XdlZ5/xfs_metadump.gz > > I hope it helps you trigger the bug during your tests. So what about the issue? Is there a fix available? Will it get upstream in the next time? I'm looking forward hearing about that topic. > >> >> (And yes, I /do/ have a lightly tested fix which I'm about to send >> out.) >> >> --D >> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-xfs" >>> in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2017-07-10 6:34 UTC | newest] Thread overview: 12+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2017-06-14 8:22 XFS Calltraces by using XFS with Ceph list 2017-06-14 12:08 ` Brian Foster 2017-06-14 13:22 ` list 2017-06-14 14:07 ` Brian Foster 2017-06-14 15:55 ` Darrick J. Wong 2017-06-15 5:55 ` list 2017-06-15 17:20 ` Emmanuel Florac 2017-06-16 5:37 ` list 2017-06-16 13:15 ` Emmanuel Florac 2017-06-16 17:46 ` Darrick J. Wong 2017-06-19 10:17 ` list 2017-07-10 6:34 ` list
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).