* Kernel BUG at dm-cache-policy-mq.c
@ 2017-03-21 13:02 Stanislas Oger
2017-03-21 16:26 ` Mike Snitzer
0 siblings, 1 reply; 5+ messages in thread
From: Stanislas Oger @ 2017-03-21 13:02 UTC (permalink / raw)
To: dm-devel; +Cc: Maza Benjamin
Hi,
We currently encounter a critical issue on a Proxmox cluster we operate,
which seems to be triggered by a bug in dm-cache ("kernel BUG at
drivers/md/dm-cache-policy-mq.c:1079!", see syslog below).
1/ Context
The Proxmox cluster uses 4.4 kernel, the VM storage is a DRBD9 cluster
on top of lvm with SSD caching. The underlaying disks are on a MegaRAID
hardware RAID.
The problem started to occur since we installed a VM (a mail server)
that performs many disk reads on many small files (~ 1 million), with
read lock using flock at each read. With the VM fully running, the IO
wait of the system is less than 1%.
2/ The problem
Randomly, without pre-fail signs, syslog reports a bug in
dm-cache-policy-mq.c (see below). A few minutes later all write
operations infinitely block. A few minutes after the node stopped to
perform write operations, the other DRBD9 nodes stop writing too. At
this point all the cluster is down. Reads can be done as usual, but
write operations are inifitinely blocking.
The only way we figured out to overcome this situation is to perform a
hard reboot of the failing node. As soon as the failing node is down,
the other nodes resume to a normal activity. When the failing node is up
again, DRBD9 performs disk resynchronization and the cluster resume
normal activity, as if nothing happened.
The bug occurred with both 4.4.35 and 4.4.40 kernels, with a frequency
of about once every 10 days.
3/ Hardware RAID info
Basics :
Model = LSI MegaRAID SAS 9271-4i
Serial Number = SK64414158
Mfg Date = 11/05/16
Revision No = 001
Version :
Firmware Package Build = 23.34.0-0019
Firmware Version = 3.460.115-6465
Bios Version = 5.50.03.0_4.17.08.00_0x06110200
NVDATA Version = 2.1507.03-0162
Boot Block Version = 2.05.00.00-0010
Bootloader Version = 07.26.26.219
Driver Name = megaraid_sas
Driver Version = 06.810.09.00-rc1
4/ Syslog trace
Mar 18 19:11:27 hyde kernel: [1567082.404669] kernel BUG at
drivers/md/dm-cache-policy-mq.c:1079!
Mar 18 19:11:27 hyde kernel: [1567082.405274] Modules linked in:
binfmt_misc ipt_REJECT nf_reject_ipv4 dm_snapshot iptable_mangle veth
ip_set ip6table_filter ip6_tables drbd_transport_tcp(O) drbd(O) softdog
nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc ocfs2_dlmfs
ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue configfs
ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp
libiscsi_tcp libiscsi scsi_transport_iscsi openvswitch nf_defrag_ipv6
xt_limit xt_conntrack xt_addrtype iptable_filter xt_nat xt_tcpudp
xt_multiport iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4
nf_nat nf_conntrack iptable_raw ip_tables x_tables nfnetlink_log
nfnetlink zfs(PO) zunicode(PO) zcommon(PO) znvpair(PO) spl(O) zavl(PO)
dm_cache_mq dm_cache dm_thin_pool dm_persistent_data dm_bio_prison
dm_bufio libcrc32c intel_rapl<4>[1567082.409411] CPU: 2 PID: 16388 Comm:
php5-fpm Tainted: P O 4.4.40-1-pve #1
Mar 18 19:11:27 hyde kernel: [1567082.409912] Hardware name: Supermicro
Super Server/X10SRi-F, BIOS 2.0 12/17/2015
Mar 18 19:11:27 hyde kernel: [1567082.410936] RIP:
0010:[<ffffffffc0376123>] [<ffffffffc0376123>]
__mq_set_clear_dirty+0x43/0x80 [dm_cache_mq]
Mar 18 19:11:27 hyde kernel: [1567082.411981] RAX: 0000000000000000 RBX:
ffff88003590c000 RCX: ffffc90014ba2ba0
Mar 18 19:11:27 hyde kernel: [1567082.413031] RBP: ffff8806b85ab8f8 R08:
0000000000000000 R09: ffff88003590c630
Mar 18 19:11:27 hyde kernel: [1567082.414147] R13: 00000000007429e9 R14:
0000000000008b7f R15: 0000000000000000
Mar 18 19:11:27 hyde kernel: [1567082.415202] CS: 0010 DS: 0000 ES:
0000 CR0: 0000000080050033
Mar 18 19:11:27 hyde kernel: [1567082.416896] ffff8806b85ab8f8
ffff88003590c080 ffff88003590c000 ffff8806b85ab920
Mar 18 19:11:27 hyde kernel: [1567082.418675] Call Trace:
Mar 18 19:11:27 hyde kernel: [1567082.420566] [<ffffffffc0578e09>]
remap_cell_to_cache_dirty+0x1d9/0x240 [dm_cache]
Mar 18 19:11:27 hyde kernel: [1567082.422520] [<ffffffffc05762a0>] ?
cache_resume+0x30/0x30 [dm_cache]
Mar 18 19:11:27 hyde kernel: [1567082.425221] [<ffffffff813ca1c0>]
generic_make_request+0x110/0x1f0
Mar 18 19:11:27 hyde kernel: [1567082.428555] [<ffffffff81190146>]
__filemap_fdatawrite_range+0xc6/0x100
Mar 18 19:11:27 hyde kernel: [1567082.431999] [<ffffffff8121097e>]
____fput+0xe/0x10
Mar 18 19:11:27 hyde kernel: [1567082.435492] Code: 08 48 8b bf 78 0d 00
00 48 8b b3 80 0d 00 00 e8 64 f6 ff ff 48 85 c0 74 12 48 3b 83 f8 00 00
00 72 09 48 3b 83 00 01 00 00 72 02 <0f> 0b 48 89 c6 48 89 df 48 89 45
e8 e8 4c ef ff ff 48 8b 45 e8
Mar 18 19:11:27 hyde kernel: [1567082.445413] Modules linked in:
binfmt_misc ipt_REJECT nf_reject_ipv4 dm_snapshot iptable_mangle veth
ip_set ip6table_filter ip6_tables drbd_transport_tcp(O) drbd(O) softdog
nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc ocfs2_dlmfs
ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue configfs
ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp
libiscsi_tcp libiscsi scsi_transport_iscsi openvswitch nf_defrag_ipv6
xt_limit xt_conntrack xt_addrtype iptable_filter xt_nat xt_tcpudp
xt_multiport iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4
nf_nat nf_conntrack iptable_raw ip_tables x_tables nfnetlink_log
nfnetlink zfs(PO) zunicode(PO) zcommon(PO) znvpair(PO) spl(O) zavl(PO)
dm_cache_mq dm_cache dm_thin_pool dm_persistent_data dm_bio_prison
dm_bufio libcrc32c intel_rapl<4>[1567082.461675] [<ffffffff81031db1>]
oops_end+0xa1/0xd0
Mar 18 19:11:27 hyde kernel: [1567082.465256] [<ffffffff810b3faf>] ?
select_idle_sibling+0xef/0x120
Mar 18 19:11:27 hyde kernel: [1567082.468782] [<ffffffffc0376123>] ?
__mq_set_clear_dirty+0x43/0x80 [dm_cache_mq]
Mar 18 19:11:27 hyde kernel: [1567082.472294] [<ffffffffc0579336>]
cache_map+0x326/0x4b0 [dm_cache]
Mar 18 19:11:27 hyde kernel: [1567082.475703] [<ffffffff813ca1c0>]
generic_make_request+0x110/0x1f0
Mar 18 19:11:27 hyde kernel: [1567082.479114] [<ffffffff81190146>]
__filemap_fdatawrite_range+0xc6/0x100
Mar 18 19:11:27 hyde kernel: [1567082.482535] [<ffffffff8121097e>]
____fput+0xe/0x10
Mar 18 19:11:27 hyde kernel: [1567082.485950] ---[ end trace
0767d58f6fa0ec61 ]---
I found a similar bug report on the <4.2 kernel
(https://www.redhat.com/archives/linux-lvm/2015-November/msg00017.html),
but it should be fixed in the 4.4 kernel.
Have you any idea of what can cause this issue?
If you need more info on the system, please ask.
Thank you,
Stanislas.
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: Kernel BUG at dm-cache-policy-mq.c
2017-03-21 13:02 Kernel BUG at dm-cache-policy-mq.c Stanislas Oger
@ 2017-03-21 16:26 ` Mike Snitzer
2017-03-21 19:46 ` Stanislas Oger
0 siblings, 1 reply; 5+ messages in thread
From: Mike Snitzer @ 2017-03-21 16:26 UTC (permalink / raw)
To: Stanislas Oger; +Cc: dm-devel, Maza Benjamin
On Tue, Mar 21 2017 at 9:02am -0400,
Stanislas Oger <stanislas.oger@gmail.com> wrote:
> Hi,
>
> We currently encounter a critical issue on a Proxmox cluster we
> operate, which seems to be triggered by a bug in dm-cache ("kernel
> BUG at drivers/md/dm-cache-policy-mq.c:1079!", see syslog below).
>
>
> 1/ Context
>
> The Proxmox cluster uses 4.4 kernel, the VM storage is a DRBD9
> cluster on top of lvm with SSD caching. The underlaying disks are on
> a MegaRAID hardware RAID.
> The problem started to occur since we installed a VM (a mail server)
> that performs many disk reads on many small files (~ 1 million),
> with read lock using flock at each read. With the VM fully running,
> the IO wait of the system is less than 1%.
>
>
> 2/ The problem
>
> Randomly, without pre-fail signs, syslog reports a bug in
> dm-cache-policy-mq.c (see below). A few minutes later all write
> operations infinitely block. A few minutes after the node stopped to
> perform write operations, the other DRBD9 nodes stop writing too. At
> this point all the cluster is down. Reads can be done as usual, but
> write operations are inifitinely blocking.
>
> The only way we figured out to overcome this situation is to perform
> a hard reboot of the failing node. As soon as the failing node is
> down, the other nodes resume to a normal activity. When the failing
> node is up again, DRBD9 performs disk resynchronization and the
> cluster resume normal activity, as if nothing happened.
>
> The bug occurred with both 4.4.35 and 4.4.40 kernels, with a
> frequency of about once every 10 days.
How large is your cache? (size of slow and fast device?)
Have you tried the smq policy? mq is no longer maintained (has been
removed and made an alias of smq, see commit 9ed84698fdda ("dm cache:
make the 'mq' policy an alias for 'smq'")).
It should be noted that dm-cache is changing significantly in 4.12
(already staged in linux-next), see:
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-4.12
The new smq code doesn't have the BUG_ON() in question.
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: Kernel BUG at dm-cache-policy-mq.c
2017-03-21 16:26 ` Mike Snitzer
@ 2017-03-21 19:46 ` Stanislas Oger
0 siblings, 0 replies; 5+ messages in thread
From: Stanislas Oger @ 2017-03-21 19:46 UTC (permalink / raw)
To: Mike Snitzer; +Cc: dm-devel, Maza Benjamin
Thank you for your reply Mike.
The fast SSD cache is 426GB large and the slow HDD storage is 3.6TB large.
Thank you for pointing out the deprecation of the mq policy, we will try
switch to smq and see what happen.
Regards.
On 21/03/2017 17:26, Mike Snitzer wrote:
> On Tue, Mar 21 2017 at 9:02am -0400,
> Stanislas Oger <stanislas.oger@gmail.com> wrote:
>
>> Hi,
>>
>> We currently encounter a critical issue on a Proxmox cluster we
>> operate, which seems to be triggered by a bug in dm-cache ("kernel
>> BUG at drivers/md/dm-cache-policy-mq.c:1079!", see syslog below).
>>
>>
>> 1/ Context
>>
>> The Proxmox cluster uses 4.4 kernel, the VM storage is a DRBD9
>> cluster on top of lvm with SSD caching. The underlaying disks are on
>> a MegaRAID hardware RAID.
>> The problem started to occur since we installed a VM (a mail server)
>> that performs many disk reads on many small files (~ 1 million),
>> with read lock using flock at each read. With the VM fully running,
>> the IO wait of the system is less than 1%.
>>
>>
>> 2/ The problem
>>
>> Randomly, without pre-fail signs, syslog reports a bug in
>> dm-cache-policy-mq.c (see below). A few minutes later all write
>> operations infinitely block. A few minutes after the node stopped to
>> perform write operations, the other DRBD9 nodes stop writing too. At
>> this point all the cluster is down. Reads can be done as usual, but
>> write operations are inifitinely blocking.
>>
>> The only way we figured out to overcome this situation is to perform
>> a hard reboot of the failing node. As soon as the failing node is
>> down, the other nodes resume to a normal activity. When the failing
>> node is up again, DRBD9 performs disk resynchronization and the
>> cluster resume normal activity, as if nothing happened.
>>
>> The bug occurred with both 4.4.35 and 4.4.40 kernels, with a
>> frequency of about once every 10 days.
> How large is your cache? (size of slow and fast device?)
>
> Have you tried the smq policy? mq is no longer maintained (has been
> removed and made an alias of smq, see commit 9ed84698fdda ("dm cache:
> make the 'mq' policy an alias for 'smq'")).
>
> It should be noted that dm-cache is changing significantly in 4.12
> (already staged in linux-next), see:
> https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-4.12
>
> The new smq code doesn't have the BUG_ON() in question.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Kernel BUG at dm-cache-policy-mq.c
@ 2017-03-20 12:15 Stanislas Oger
0 siblings, 0 replies; 5+ messages in thread
From: Stanislas Oger @ 2017-03-20 12:15 UTC (permalink / raw)
To: dm-devel; +Cc: Maza Benjamin
Hi,
We currently encounter a critical issue on a Proxmox cluster we operate,
which seems to be triggered by a bug in dm-cache ("kernel BUG at
drivers/md/dm-cache-policy-mq.c:1079!", see syslog below).
1/ Context
The Proxmox cluster uses 4.4 kernel, the VM storage is a DRBD9 cluster
on top of lvm with SSD caching. The underlaying disks are on a MegaRAID
hardware RAID.
The problem started to occur since we installed a VM (a mail server)
that performs many disk reads on many small files (~ 1 million), with
read lock using flock at each read. With the VM fully running, the IO
wait of the system is less than 1%.
2/ The problem
Randomly, without pre-fail signs, syslog reports a bug in
dm-cache-policy-mq.c (see below). A few minutes later all write
operations infinitely block. A few minutes after the node stopped to
perform write operations, the other DRBD9 nodes stop writing too. At
this point all the cluster is down. Reads can be done as usual, but
write operations are inifitinely blocking.
The only way we figured out to overcome this situation is to perform a
hard reboot of the failing node. As soon as the failing node is down,
the other nodes resume to a normal activity. When the failing node is up
again, DRBD9 performs disk resynchronization and the cluster resume
normal activity, as if nothing happened.
The bug occurred with both 4.4.35 and 4.4.40 kernels, with a frequency
of about once every 10 days.
3/ Hardware RAID info
Basics :
Model = LSI MegaRAID SAS 9271-4i
Serial Number = SK64414158
Mfg Date = 11/05/16
Revision No = 001
Version :
Firmware Package Build = 23.34.0-0019
Firmware Version = 3.460.115-6465
Bios Version = 5.50.03.0_4.17.08.00_0x06110200
NVDATA Version = 2.1507.03-0162
Boot Block Version = 2.05.00.00-0010
Bootloader Version = 07.26.26.219
Driver Name = megaraid_sas
Driver Version = 06.810.09.00-rc1
4/ Syslog trace
Mar 18 19:11:27 hyde kernel: [1567082.404669] kernel BUG at
drivers/md/dm-cache-policy-mq.c:1079!
Mar 18 19:11:27 hyde kernel: [1567082.405274] Modules linked in:
binfmt_misc ipt_REJECT nf_reject_ipv4 dm_snapshot iptable_mangle veth
ip_set ip6table_filter ip6_tables drbd_transport_tcp(O) drbd(O) softdog
nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc ocfs2_dlmfs
ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue configfs
ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp
libiscsi_tcp libiscsi scsi_transport_iscsi openvswitch nf_defrag_ipv6
xt_limit xt_conntrack xt_addrtype iptable_filter xt_nat xt_tcpudp
xt_multiport iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4
nf_nat nf_conntrack iptable_raw ip_tables x_tables nfnetlink_log
nfnetlink zfs(PO) zunicode(PO) zcommon(PO) znvpair(PO) spl(O) zavl(PO)
dm_cache_mq dm_cache dm_thin_pool dm_persistent_data dm_bio_prison
dm_bufio libcrc32c intel_rapl<4>[1567082.409411] CPU: 2 PID: 16388 Comm:
php5-fpm Tainted: P O 4.4.40-1-pve #1
Mar 18 19:11:27 hyde kernel: [1567082.409912] Hardware name: Supermicro
Super Server/X10SRi-F, BIOS 2.0 12/17/2015
Mar 18 19:11:27 hyde kernel: [1567082.410936] RIP:
0010:[<ffffffffc0376123>] [<ffffffffc0376123>]
__mq_set_clear_dirty+0x43/0x80 [dm_cache_mq]
Mar 18 19:11:27 hyde kernel: [1567082.411981] RAX: 0000000000000000 RBX:
ffff88003590c000 RCX: ffffc90014ba2ba0
Mar 18 19:11:27 hyde kernel: [1567082.413031] RBP: ffff8806b85ab8f8 R08:
0000000000000000 R09: ffff88003590c630
Mar 18 19:11:27 hyde kernel: [1567082.414147] R13: 00000000007429e9 R14:
0000000000008b7f R15: 0000000000000000
Mar 18 19:11:27 hyde kernel: [1567082.415202] CS: 0010 DS: 0000 ES:
0000 CR0: 0000000080050033
Mar 18 19:11:27 hyde kernel: [1567082.416896] ffff8806b85ab8f8
ffff88003590c080 ffff88003590c000 ffff8806b85ab920
Mar 18 19:11:27 hyde kernel: [1567082.418675] Call Trace:
Mar 18 19:11:27 hyde kernel: [1567082.420566] [<ffffffffc0578e09>]
remap_cell_to_cache_dirty+0x1d9/0x240 [dm_cache]
Mar 18 19:11:27 hyde kernel: [1567082.422520] [<ffffffffc05762a0>] ?
cache_resume+0x30/0x30 [dm_cache]
Mar 18 19:11:27 hyde kernel: [1567082.425221] [<ffffffff813ca1c0>]
generic_make_request+0x110/0x1f0
Mar 18 19:11:27 hyde kernel: [1567082.428555] [<ffffffff81190146>]
__filemap_fdatawrite_range+0xc6/0x100
Mar 18 19:11:27 hyde kernel: [1567082.431999] [<ffffffff8121097e>]
____fput+0xe/0x10
Mar 18 19:11:27 hyde kernel: [1567082.435492] Code: 08 48 8b bf 78 0d 00
00 48 8b b3 80 0d 00 00 e8 64 f6 ff ff 48 85 c0 74 12 48 3b 83 f8 00 00
00 72 09 48 3b 83 00 01 00 00 72 02 <0f> 0b 48 89 c6 48 89 df 48 89 45
e8 e8 4c ef ff ff 48 8b 45 e8
Mar 18 19:11:27 hyde kernel: [1567082.445413] Modules linked in:
binfmt_misc ipt_REJECT nf_reject_ipv4 dm_snapshot iptable_mangle veth
ip_set ip6table_filter ip6_tables drbd_transport_tcp(O) drbd(O) softdog
nfsd auth_rpcgss nfs_acl nfs lockd grace fscache sunrpc ocfs2_dlmfs
ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue configfs
ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp
libiscsi_tcp libiscsi scsi_transport_iscsi openvswitch nf_defrag_ipv6
xt_limit xt_conntrack xt_addrtype iptable_filter xt_nat xt_tcpudp
xt_multiport iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4
nf_nat nf_conntrack iptable_raw ip_tables x_tables nfnetlink_log
nfnetlink zfs(PO) zunicode(PO) zcommon(PO) znvpair(PO) spl(O) zavl(PO)
dm_cache_mq dm_cache dm_thin_pool dm_persistent_data dm_bio_prison
dm_bufio libcrc32c intel_rapl<4>[1567082.461675] [<ffffffff81031db1>]
oops_end+0xa1/0xd0
Mar 18 19:11:27 hyde kernel: [1567082.465256] [<ffffffff810b3faf>] ?
select_idle_sibling+0xef/0x120
Mar 18 19:11:27 hyde kernel: [1567082.468782] [<ffffffffc0376123>] ?
__mq_set_clear_dirty+0x43/0x80 [dm_cache_mq]
Mar 18 19:11:27 hyde kernel: [1567082.472294] [<ffffffffc0579336>]
cache_map+0x326/0x4b0 [dm_cache]
Mar 18 19:11:27 hyde kernel: [1567082.475703] [<ffffffff813ca1c0>]
generic_make_request+0x110/0x1f0
Mar 18 19:11:27 hyde kernel: [1567082.479114] [<ffffffff81190146>]
__filemap_fdatawrite_range+0xc6/0x100
Mar 18 19:11:27 hyde kernel: [1567082.482535] [<ffffffff8121097e>]
____fput+0xe/0x10
Mar 18 19:11:27 hyde kernel: [1567082.485950] ---[ end trace
0767d58f6fa0ec61 ]---
I found a similar bug report on the <4.2 kernel
(https://www.redhat.com/archives/linux-lvm/2015-November/msg00017.html),
but it should be fixed in the 4.4 kernel.
Have you any idea of what can cause this issue?
If you need more info on the system, please ask.
Thank you,
Stanislas.
^ permalink raw reply [flat|nested] 5+ messages in thread* [linux-lvm] Kernel BUG at dm-cache-policy-mq.c
@ 2015-11-19 9:32 Ciprian Hacman
2015-11-19 15:49 ` Mike Snitzer
0 siblings, 1 reply; 5+ messages in thread
From: Ciprian Hacman @ 2015-11-19 9:32 UTC (permalink / raw)
To: LVM general discussion and development
[-- Attachment #1: Type: text/plain, Size: 6750 bytes --]
Hi,
One more issue from me. As I said in my previous email, we are configuring
lvm with SSD caching and EBS volumes on some of our boxes in AWS. The OS
for those nodes is Ubuntu 15.10 (4.2.0-16-generic).
We already had 2 nodes down and seems to be related to the lvm caching
part. On one of the nodes we found this in the logs:
Nov 17 17:03:26 localhost kernel: [1650439.548785] ------------[ cut here
]------------
Nov 17 17:03:26 localhost kernel: [1650439.552225] kernel BUG at
/build/linux-AxjFAn/linux-4.2.0/drivers/md/dm-cache-policy-mq.c:1079!
Nov 17 17:03:26 localhost kernel: [1650439.552561] invalid opcode: 0000
[#1] SMP
Nov 17 17:03:26 localhost kernel: [1650439.552561] Modules linked in: isofs
binfmt_misc xt_CHECKSUM iptable_mangle ipt_MASQUERADE
nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc iptable_filter
ip_tables x_tables dm_cache_mq dm_cache dm_persistent_data dm_bio_prison
dm_bufio libcrc32c ppdev xen_fbfront syscopyarea sysfillrect sysimgblt
fb_sys_fops serio_raw parport_pc parport autofs4 raid10 raid456
async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq
raid1 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel
raid0 aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd
psmouse floppy
Nov 17 17:03:26 localhost kernel: [1650439.552561] CPU: 1 PID: 68058 Comm:
java Not tainted 4.2.0-16-generic #19-Ubuntu
Nov 17 17:03:26 localhost kernel: [1650439.552561] Hardware name: Xen HVM
domU, BIOS 4.2.amazon 05/06/2015
Nov 17 17:03:26 localhost kernel: [1650439.552561] task: ffff880190241b80
ti: ffff8806f3cf4000 task.ti: ffff8806f3cf4000
Nov 17 17:03:26 localhost kernel: [1650439.552561] RIP:
0010:[<ffffffffc0182257>] [<ffffffffc0182257>]
__mq_set_clear_dirty+0x47/0x80 [dm_cache_mq]
Nov 17 17:03:26 localhost kernel: [1650439.552561] RSP:
0018:ffff8806f3cf7730 EFLAGS: 00010246
Nov 17 17:03:26 localhost kernel: [1650439.552561] RAX: 0000000000000000
RBX: ffff88076a236080 RCX: ffffc90020f6aff8
Nov 17 17:03:26 localhost kernel: [1650439.552561] RDX: 0000000000f7b83e
RSI: ffffc9001fd39000 RDI: 0000000000000016
Nov 17 17:03:26 localhost kernel: [1650439.552561] RBP: ffff8806f3cf7748
R08: 0000000000000000 R09: ffff8801adb6c7c8
Nov 17 17:03:26 localhost kernel: [1650439.552561] R10: ffff88032fd31bb0
R11: ffff88076a22c858 R12: ffff88076a236000
Nov 17 17:03:26 localhost kernel: [1650439.552561] R13: 0000000000000001
R14: 000000000045c6ae R15: 0000000000000000
Nov 17 17:03:26 localhost kernel: [1650439.552561] FS:
00007fccc4b27700(0000) GS:ffff88076f640000(0000) knlGS:0000000000000000
Nov 17 17:03:26 localhost kernel: [1650439.552561] CS: 0010 DS: 0000 ES:
0000 CR0: 0000000080050033
Nov 17 17:03:26 localhost kernel: [1650439.552561] CR2: 00007fce83a55000
CR3: 00000005b3d2b000 CR4: 00000000001406e0
Nov 17 17:03:26 localhost kernel: [1650439.552561] Stack:
Nov 17 17:03:26 localhost kernel: [1650439.552561] ffff88076a236080
ffff88076a236000 0000000000f7b83e ffff8806f3cf7778
Nov 17 17:03:26 localhost kernel: [1650439.552561] ffffffffc0182317
0000000000000000 000000000045c6ae ffff880476c014e0
Nov 17 17:03:26 localhost kernel: [1650439.552561] ffff88076744f800
ffff8806f3cf7788 ffffffffc01a9862 ffff8806f3cf7818
Nov 17 17:03:26 localhost kernel: [1650439.552561] Call Trace:
Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffffc0182317>]
mq_set_dirty+0x37/0x50 [dm_cache_mq]
Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffffc01a9862>]
set_dirty+0x32/0x40 [dm_cache]
Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffffc01ab3c9>]
remap_cell_to_cache_dirty+0x1d9/0x240 [dm_cache]
Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffffc01ab900>]
cache_map+0x330/0x4d0 [dm_cache]
Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffffc01a8eb0>] ?
cache_resume+0x30/0x30 [dm_cache]
Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8166b2ee>]
__map_bio+0x3e/0x100
Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8166d235>]
__split_and_process_bio+0x285/0x3f0
Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8166d40d>]
dm_make_request+0x6d/0xc0
Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff813952a6>]
generic_make_request+0xd6/0x110
Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff810c3d61>] ?
__raw_callee_save___pv_queued_spin_unlock+0x11/0x20
Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff81395356>]
submit_bio+0x76/0x170
Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8138f51b>] ?
__bio_add_page.part.16+0x10b/0x270
Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8128c311>]
ext4_io_submit+0x31/0x50
Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8128c4c8>]
ext4_bio_write_page+0x168/0x410
Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff81283351>]
mpage_submit_page+0x61/0x80
Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff812835d6>]
mpage_map_and_submit_buffers+0x156/0x290
Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff81288874>]
ext4_writepages+0x624/0xce0
Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff811903be>]
do_writepages+0x1e/0x30
Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8118335c>]
__filemap_fdatawrite_range+0xcc/0x100
Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8118349a>]
filemap_write_and_wait_range+0x2a/0x70
Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8127f831>]
ext4_sync_file+0xe1/0x2f0
Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8122fc9b>]
vfs_fsync_range+0x4b/0xb0
Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8122fd5d>]
do_fsync+0x3d/0x70
Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff81230023>]
SyS_fdatasync+0x13/0x20
Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff817ef9f2>]
entry_SYSCALL_64_fastpath+0x16/0x75
Nov 17 17:03:26 localhost kernel: [1650439.552561] Code: 89 f2 49 8b b4 24
80 0d 00 00 e8 c5 f5 ff ff 48 85 c0 74 17 49 3b 84 24 f8 00 00 00 48 89 c3
72 0a 49 3b 84 24 00 01 00 00 72 02 <0f> 0b 48 89 c6 4c 89 e7 41 83 e5 01
e8 08 ef ff ff 0f b6 43 28
Nov 17 17:03:26 localhost kernel: [1650439.552561] RIP
[<ffffffffc0182257>] __mq_set_clear_dirty+0x47/0x80 [dm_cache_mq]
Nov 17 17:03:26 localhost kernel: [1650439.552561] RSP <ffff8806f3cf7730>
Nov 17 17:03:26 localhost kernel: [1650439.740854] ---[ end trace
98483c1d54cc426e ]---
Is this something that has been seen before?
Would switching to RHEL/CentOS 7 make any difference?
Thanks,
Ciprian Hacman
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
[-- Attachment #2: Type: text/html, Size: 10900 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Kernel BUG at dm-cache-policy-mq.c
2015-11-19 9:32 [linux-lvm] " Ciprian Hacman
@ 2015-11-19 15:49 ` Mike Snitzer
0 siblings, 0 replies; 5+ messages in thread
From: Mike Snitzer @ 2015-11-19 15:49 UTC (permalink / raw)
To: Ciprian Hacman; +Cc: dm-devel, ejt, LVM general discussion and development
On Thu, Nov 19 2015 at 4:32am -0500,
Ciprian Hacman <ciprian.hacman@sematext.com> wrote:
> Hi,
>
> One more issue from me. As I said in my previous email, we are configuring
> lvm with SSD caching and EBS volumes on some of our boxes in AWS. The OS
> for those nodes is Ubuntu 15.10 (4.2.0-16-generic).
>
> We already had 2 nodes down and seems to be related to the lvm caching
> part. On one of the nodes we found this in the logs:
<snip>
Please send any kernel issues to dm-devel@redhat.com in the future.
> Nov 17 17:03:26 localhost kernel: [1650439.548785] ------------[ cut here
> ]------------
> Nov 17 17:03:26 localhost kernel: [1650439.552225] kernel BUG at
> /build/linux-AxjFAn/linux-4.2.0/drivers/md/dm-cache-policy-mq.c:1079!
> Nov 17 17:03:26 localhost kernel: [1650439.552561] invalid opcode: 0000
> [#1] SMP
> Nov 17 17:03:26 localhost kernel: [1650439.552561] Modules linked in: isofs
> binfmt_misc xt_CHECKSUM iptable_mangle ipt_MASQUERADE
> nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
> nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc iptable_filter
> ip_tables x_tables dm_cache_mq dm_cache dm_persistent_data dm_bio_prison
> dm_bufio libcrc32c ppdev xen_fbfront syscopyarea sysfillrect sysimgblt
> fb_sys_fops serio_raw parport_pc parport autofs4 raid10 raid456
> async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq
> raid1 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel
> raid0 aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd
> psmouse floppy
> Nov 17 17:03:26 localhost kernel: [1650439.552561] CPU: 1 PID: 68058 Comm:
> java Not tainted 4.2.0-16-generic #19-Ubuntu
> Nov 17 17:03:26 localhost kernel: [1650439.552561] Hardware name: Xen HVM
> domU, BIOS 4.2.amazon 05/06/2015
> Nov 17 17:03:26 localhost kernel: [1650439.552561] task: ffff880190241b80
> ti: ffff8806f3cf4000 task.ti: ffff8806f3cf4000
> Nov 17 17:03:26 localhost kernel: [1650439.552561] RIP:
> 0010:[<ffffffffc0182257>] [<ffffffffc0182257>]
> __mq_set_clear_dirty+0x47/0x80 [dm_cache_mq]
> Nov 17 17:03:26 localhost kernel: [1650439.552561] RSP:
> 0018:ffff8806f3cf7730 EFLAGS: 00010246
> Nov 17 17:03:26 localhost kernel: [1650439.552561] RAX: 0000000000000000
> RBX: ffff88076a236080 RCX: ffffc90020f6aff8
> Nov 17 17:03:26 localhost kernel: [1650439.552561] RDX: 0000000000f7b83e
> RSI: ffffc9001fd39000 RDI: 0000000000000016
> Nov 17 17:03:26 localhost kernel: [1650439.552561] RBP: ffff8806f3cf7748
> R08: 0000000000000000 R09: ffff8801adb6c7c8
> Nov 17 17:03:26 localhost kernel: [1650439.552561] R10: ffff88032fd31bb0
> R11: ffff88076a22c858 R12: ffff88076a236000
> Nov 17 17:03:26 localhost kernel: [1650439.552561] R13: 0000000000000001
> R14: 000000000045c6ae R15: 0000000000000000
> Nov 17 17:03:26 localhost kernel: [1650439.552561] FS:
> 00007fccc4b27700(0000) GS:ffff88076f640000(0000) knlGS:0000000000000000
> Nov 17 17:03:26 localhost kernel: [1650439.552561] CS: 0010 DS: 0000 ES:
> 0000 CR0: 0000000080050033
> Nov 17 17:03:26 localhost kernel: [1650439.552561] CR2: 00007fce83a55000
> CR3: 00000005b3d2b000 CR4: 00000000001406e0
> Nov 17 17:03:26 localhost kernel: [1650439.552561] Stack:
> Nov 17 17:03:26 localhost kernel: [1650439.552561] ffff88076a236080
> ffff88076a236000 0000000000f7b83e ffff8806f3cf7778
> Nov 17 17:03:26 localhost kernel: [1650439.552561] ffffffffc0182317
> 0000000000000000 000000000045c6ae ffff880476c014e0
> Nov 17 17:03:26 localhost kernel: [1650439.552561] ffff88076744f800
> ffff8806f3cf7788 ffffffffc01a9862 ffff8806f3cf7818
> Nov 17 17:03:26 localhost kernel: [1650439.552561] Call Trace:
> Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffffc0182317>]
> mq_set_dirty+0x37/0x50 [dm_cache_mq]
> Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffffc01a9862>]
> set_dirty+0x32/0x40 [dm_cache]
> Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffffc01ab3c9>]
> remap_cell_to_cache_dirty+0x1d9/0x240 [dm_cache]
> Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffffc01ab900>]
> cache_map+0x330/0x4d0 [dm_cache]
> Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffffc01a8eb0>] ?
> cache_resume+0x30/0x30 [dm_cache]
> Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8166b2ee>]
> __map_bio+0x3e/0x100
> Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8166d235>]
> __split_and_process_bio+0x285/0x3f0
> Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8166d40d>]
> dm_make_request+0x6d/0xc0
> Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff813952a6>]
> generic_make_request+0xd6/0x110
> Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff810c3d61>] ?
> __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
> Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff81395356>]
> submit_bio+0x76/0x170
> Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8138f51b>] ?
> __bio_add_page.part.16+0x10b/0x270
> Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8128c311>]
> ext4_io_submit+0x31/0x50
> Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8128c4c8>]
> ext4_bio_write_page+0x168/0x410
> Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff81283351>]
> mpage_submit_page+0x61/0x80
> Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff812835d6>]
> mpage_map_and_submit_buffers+0x156/0x290
> Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff81288874>]
> ext4_writepages+0x624/0xce0
> Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff811903be>]
> do_writepages+0x1e/0x30
> Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8118335c>]
> __filemap_fdatawrite_range+0xcc/0x100
> Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8118349a>]
> filemap_write_and_wait_range+0x2a/0x70
> Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8127f831>]
> ext4_sync_file+0xe1/0x2f0
> Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8122fc9b>]
> vfs_fsync_range+0x4b/0xb0
> Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff8122fd5d>]
> do_fsync+0x3d/0x70
> Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff81230023>]
> SyS_fdatasync+0x13/0x20
> Nov 17 17:03:26 localhost kernel: [1650439.552561] [<ffffffff817ef9f2>]
> entry_SYSCALL_64_fastpath+0x16/0x75
> Nov 17 17:03:26 localhost kernel: [1650439.552561] Code: 89 f2 49 8b b4 24
> 80 0d 00 00 e8 c5 f5 ff ff 48 85 c0 74 17 49 3b 84 24 f8 00 00 00 48 89 c3
> 72 0a 49 3b 84 24 00 01 00 00 72 02 <0f> 0b 48 89 c6 4c 89 e7 41 83 e5 01
> e8 08 ef ff ff 0f b6 43 28
> Nov 17 17:03:26 localhost kernel: [1650439.552561] RIP
> [<ffffffffc0182257>] __mq_set_clear_dirty+0x47/0x80 [dm_cache_mq]
> Nov 17 17:03:26 localhost kernel: [1650439.552561] RSP <ffff8806f3cf7730>
> Nov 17 17:03:26 localhost kernel: [1650439.740854] ---[ end trace
> 98483c1d54cc426e ]---
>
>
> Is this something that has been seen before?
> Would switching to RHEL/CentOS 7 make any difference?
AFAIK, this issue was already fixed with the 4.2 release, via commit
fb4100ae7f31 ("dm cache: fix race when issuing a POLICY_REPLACE
operation")
But if ubuntu's kernel trully is based on the upstream 4.2 kernel then
maybe there is something else going on...
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2017-03-21 19:46 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-03-21 13:02 Kernel BUG at dm-cache-policy-mq.c Stanislas Oger
2017-03-21 16:26 ` Mike Snitzer
2017-03-21 19:46 ` Stanislas Oger
-- strict thread matches above, loose matches on Subject: below --
2017-03-20 12:15 Stanislas Oger
2015-11-19 9:32 [linux-lvm] " Ciprian Hacman
2015-11-19 15:49 ` Mike Snitzer
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.