linux-xfs.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: list@jonas-server.de
To: linux-xfs@vger.kernel.org
Subject: XFS Calltraces by using XFS with Ceph
Date: Wed, 14 Jun 2017 10:22:38 +0200	[thread overview]
Message-ID: <ff5af7a48b502752faac6f2bbc1c3bb7@jonas-server.de> (raw)

Hello guys,

we have currently an issue with our ceph setup based on XFS. Sometimes 
some nodes are dying with high load with this calltrace in dmesg:

[Tue Jun 13 13:18:48 2017] BUG: unable to handle kernel NULL pointer 
dereference at 00000000000000a0
[Tue Jun 13 13:18:48 2017] IP: [<ffffffffc06555a0>] 
xfs_da3_node_read+0x30/0xb0 [xfs]
[Tue Jun 13 13:18:48 2017] PGD 0
[Tue Jun 13 13:18:48 2017] Oops: 0000 [#1] SMP
[Tue Jun 13 13:18:48 2017] Modules linked in: cpuid arc4 md4 nls_utf8 
cifs fscache nfnetlink_queue nfnetlink xt_CHECKSUM xt_nat iptable_nat 
nf_nat_ipv4 xt_NFQUEUE xt_CLASSIFY ip6table_mangle dccp_diag dccp 
tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag veth 
dummy bridge stp llc ebtable_filter ebtables iptable_mangle xt_CT 
iptable_raw nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables 
xt_tcpudp nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack ip6table_filter 
ip6_tables x_tables xfs ipmi_devintf dcdbas x86_pkg_temp_thermal 
intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul 
ghash_clmulni_intel aesni_intel ipmi_ssif aes_x86_64 lrw gf128mul 
glue_helper ablk_helper cryptd sb_edac edac_core input_leds joydev 
lpc_ich ioatdma shpchp 8250_fintek ipmi_si ipmi_msghandler acpi_pad 
acpi_power_meter
[Tue Jun 13 13:18:48 2017]  mac_hid vhost_net vhost macvtap macvlan 
kvm_intel kvm irqbypass cdc_ether nf_nat_ftp tcp_htcp nf_nat_pptp 
nf_nat_proto_gre nf_conntrack_ftp bonding nf_nat_sip nf_conntrack_sip 
nf_nat nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack usbnet mii 
lp parport autofs4 btrfs raid456 async_raid6_recov async_memcpy async_pq 
async_xor async_tx xor raid6_pq libcrc32c raid0 multipath linear raid10 
raid1 hid_generic usbhid hid ixgbe igb vxlan ip6_udp_tunnel ahci dca 
udp_tunnel libahci i2c_algo_bit ptp megaraid_sas pps_core mdio wmi fjes
[Tue Jun 13 13:18:48 2017] CPU: 3 PID: 3844 Comm: tp_fstore_op Not 
tainted 4.4.0-75-generic #96-Ubuntu
[Tue Jun 13 13:18:48 2017] Hardware name: Dell Inc. PowerEdge 
R720/0XH7F2, BIOS 2.5.4 01/22/2016
[Tue Jun 13 13:18:48 2017] task: ffff881feda65400 ti: ffff883fbda08000 
task.ti: ffff883fbda08000
[Tue Jun 13 13:18:48 2017] RIP: 0010:[<ffffffffc06555a0>]  
[<ffffffffc06555a0>] xfs_da3_node_read+0x30/0xb0 [xfs]
[Tue Jun 13 13:18:48 2017] RSP: 0018:ffff883fbda0bc88  EFLAGS: 00010286
[Tue Jun 13 13:18:48 2017] RAX: 0000000000000000 RBX: ffff8801102c5050 
RCX: 0000000000000001
[Tue Jun 13 13:18:48 2017] RDX: 0000000000000000 RSI: 0000000000000000 
RDI: ffff883fbda0bc38
[Tue Jun 13 13:18:48 2017] RBP: ffff883fbda0bca8 R08: 0000000000000001 
R09: fffffffffffffffe
[Tue Jun 13 13:18:48 2017] R10: ffff880007374ae0 R11: 0000000000000001 
R12: ffff883fbda0bcd8
[Tue Jun 13 13:18:48 2017] R13: ffff880035ac4c80 R14: 0000000000000001 
R15: 000000008b1f4885
[Tue Jun 13 13:18:48 2017] FS:  00007fc574607700(0000) 
GS:ffff883fff040000(0000) knlGS:0000000000000000
[Tue Jun 13 13:18:48 2017] CS:  0010 DS: 0000 ES: 0000 CR0: 
0000000080050033
[Tue Jun 13 13:18:48 2017] CR2: 00000000000000a0 CR3: 0000003fd828d000 
CR4: 00000000001426e0
[Tue Jun 13 13:18:48 2017] Stack:
[Tue Jun 13 13:18:48 2017]  ffffffffc06b4b50 ffffffffc0695ecc 
ffff883fbda0bde0 0000000000000001
[Tue Jun 13 13:18:48 2017]  ffff883fbda0bd20 ffffffffc06718b3 
0000000300000008 ffff880e99b44010
[Tue Jun 13 13:18:48 2017]  00000000360c65a8 ffff88270f80b900 
0000000000000000 0000000000000000
[Tue Jun 13 13:18:48 2017] Call Trace:
[Tue Jun 13 13:18:48 2017]  [<ffffffffc0695ecc>] ? 
xfs_trans_roll+0x2c/0x50 [xfs]
[Tue Jun 13 13:18:48 2017]  [<ffffffffc06718b3>] 
xfs_attr3_node_inactive+0x183/0x220 [xfs]
[Tue Jun 13 13:18:48 2017]  [<ffffffffc06718f9>] 
xfs_attr3_node_inactive+0x1c9/0x220 [xfs]
[Tue Jun 13 13:18:48 2017]  [<ffffffffc06719fc>] 
xfs_attr3_root_inactive+0xac/0x100 [xfs]
[Tue Jun 13 13:18:48 2017]  [<ffffffffc0671b9c>] 
xfs_attr_inactive+0x14c/0x1a0 [xfs]
[Tue Jun 13 13:18:48 2017]  [<ffffffffc068bda5>] xfs_inactive+0x85/0x120 
[xfs]
[Tue Jun 13 13:18:48 2017]  [<ffffffffc06912f5>] 
xfs_fs_evict_inode+0xa5/0x100 [xfs]
[Tue Jun 13 13:18:48 2017]  [<ffffffff8122a90e>] evict+0xbe/0x190
[Tue Jun 13 13:18:48 2017]  [<ffffffff8122abf1>] iput+0x1c1/0x240
[Tue Jun 13 13:18:48 2017]  [<ffffffff8121f6b9>] do_unlinkat+0x199/0x2d0
[Tue Jun 13 13:18:48 2017]  [<ffffffff81220256>] SyS_unlink+0x16/0x20
[Tue Jun 13 13:18:48 2017]  [<ffffffff8183b972>] 
entry_SYSCALL_64_fastpath+0x16/0x71
[Tue Jun 13 13:18:48 2017] Code: 55 48 89 e5 41 54 53 4d 89 c4 48 89 fb 
48 83 ec 10 48 c7 04 24 50 4b 6b c0 e8 dd fe ff ff 85 c0 75 46 48 85 db 
74 41 49 8b 34 24 <48> 8b 96 a0 00 00 00 0f b7 52 08 66 c1 c2 08 66 81 
fa be 3e 74
[Tue Jun 13 13:18:48 2017] RIP  [<ffffffffc06555a0>] 
xfs_da3_node_read+0x30/0xb0 [xfs]
[Tue Jun 13 13:18:48 2017]  RSP <ffff883fbda0bc88>
[Tue Jun 13 13:18:48 2017] CR2: 00000000000000a0
[Tue Jun 13 13:18:48 2017] ---[ end trace 5470d0d55cacb4ef ]---

The ceph OSD running on this server has then the issue that it can not 
reach any other osd in the pool.

  -1043> 2017-06-13 13:24:00.917597 7fc539a72700  0 -- 
192.168.14.19:6827/3389 >> 192.168.14.7:6805/3658 pipe(0x558219846000 
sd=23 :6827
s=0 pgs=0 cs=0 l=0 c=0x55821a330400).accept connect_seq 7 vs existing 7 
state standby
  -1042> 2017-06-13 13:24:00.918433 7fc539a72700  0 -- 
192.168.14.19:6827/3389 >> 192.168.14.7:6805/3658 pipe(0x558219846000 
sd=23 :6827
s=0 pgs=0 cs=0 l=0 c=0x55821a330400).accept connect_seq 8 vs existing 7 
state standby
  -1041> 2017-06-13 13:24:03.654983 7fc4dd21d700  0 -- 
192.168.14.19:6825/3389 >> :/0 pipe(0x5581fa6ba000 sd=524 :6825 s=0 
pgs=0 cs=0 l=0
  c=0x55820a9e5000).accept failed to getpeername (107) Transport endpoint 
is not connected


There are a lot more of these messages. Does any of you have the same 
issue? We are running Ubuntu 16.04 with kernel 4.4.0-75.96.

Best regards,
Jonas

             reply	other threads:[~2017-06-14  8:31 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-06-14  8:22 list [this message]
2017-06-14 12:08 ` XFS Calltraces by using XFS with Ceph Brian Foster
2017-06-14 13:22   ` list
2017-06-14 14:07     ` Brian Foster
2017-06-14 15:55       ` Darrick J. Wong
2017-06-15  5:55         ` list
2017-06-15 17:20           ` Emmanuel Florac
2017-06-16  5:37             ` list
2017-06-16 13:15               ` Emmanuel Florac
2017-06-16 17:46               ` Darrick J. Wong
2017-06-19 10:17                 ` list
2017-07-10  6:34                   ` list

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=ff5af7a48b502752faac6f2bbc1c3bb7@jonas-server.de \
    --to=list@jonas-server.de \
    --cc=linux-xfs@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).