From: Sasha Levin <sashal@kernel.org>
To: cluster-devel.redhat.com
Subject: [Cluster-devel] [PATCH AUTOSEL 4.19 02/59] fs: dlm: filter user dlm messages for kernel locks
Date: Mon, 17 Jan 2022 21:46:03 -0500 [thread overview]
Message-ID: <20220118024701.1952911-2-sashal@kernel.org> (raw)
In-Reply-To: <20220118024701.1952911-1-sashal@kernel.org>
From: Alexander Aring <aahringo@redhat.com>
[ Upstream commit 6c2e3bf68f3e5e5a647aa52be246d5f552d7496d ]
This patch fixes the following crash by receiving a invalid message:
[ 160.672220] ==================================================================
[ 160.676206] BUG: KASAN: user-memory-access in dlm_user_add_ast+0xc3/0x370
[ 160.679659] Read of size 8 at addr 00000000deadbeef by task kworker/u32:13/319
[ 160.681447]
[ 160.681824] CPU: 10 PID: 319 Comm: kworker/u32:13 Not tainted 5.14.0-rc2+ #399
[ 160.683472] Hardware name: Red Hat KVM/RHEL-AV, BIOS 1.14.0-1.module+el8.6.0+12648+6ede71a5 04/01/2014
[ 160.685574] Workqueue: dlm_recv process_recv_sockets
[ 160.686721] Call Trace:
[ 160.687310] dump_stack_lvl+0x56/0x6f
[ 160.688169] ? dlm_user_add_ast+0xc3/0x370
[ 160.689116] kasan_report.cold.14+0x116/0x11b
[ 160.690138] ? dlm_user_add_ast+0xc3/0x370
[ 160.690832] dlm_user_add_ast+0xc3/0x370
[ 160.691502] _receive_unlock_reply+0x103/0x170
[ 160.692241] _receive_message+0x11df/0x1ec0
[ 160.692926] ? rcu_read_lock_sched_held+0xa1/0xd0
[ 160.693700] ? rcu_read_lock_bh_held+0xb0/0xb0
[ 160.694427] ? lock_acquire+0x175/0x400
[ 160.695058] ? do_purge.isra.51+0x200/0x200
[ 160.695744] ? lock_acquired+0x360/0x5d0
[ 160.696400] ? lock_contended+0x6a0/0x6a0
[ 160.697055] ? lock_release+0x21d/0x5e0
[ 160.697686] ? lock_is_held_type+0xe0/0x110
[ 160.698352] ? lock_is_held_type+0xe0/0x110
[ 160.699026] ? ___might_sleep+0x1cc/0x1e0
[ 160.699698] ? dlm_wait_requestqueue+0x94/0x140
[ 160.700451] ? dlm_process_requestqueue+0x240/0x240
[ 160.701249] ? down_write_killable+0x2b0/0x2b0
[ 160.701988] ? do_raw_spin_unlock+0xa2/0x130
[ 160.702690] dlm_receive_buffer+0x1a5/0x210
[ 160.703385] dlm_process_incoming_buffer+0x726/0x9f0
[ 160.704210] receive_from_sock+0x1c0/0x3b0
[ 160.704886] ? dlm_tcp_shutdown+0x30/0x30
[ 160.705561] ? lock_acquire+0x175/0x400
[ 160.706197] ? rcu_read_lock_sched_held+0xa1/0xd0
[ 160.706941] ? rcu_read_lock_bh_held+0xb0/0xb0
[ 160.707681] process_recv_sockets+0x32/0x40
[ 160.708366] process_one_work+0x55e/0xad0
[ 160.709045] ? pwq_dec_nr_in_flight+0x110/0x110
[ 160.709820] worker_thread+0x65/0x5e0
[ 160.710423] ? process_one_work+0xad0/0xad0
[ 160.711087] kthread+0x1ed/0x220
[ 160.711628] ? set_kthread_struct+0x80/0x80
[ 160.712314] ret_from_fork+0x22/0x30
The issue is that we received a DLM message for a user lock but the
destination lock is a kernel lock. Note that the address which is trying
to derefence is 00000000deadbeef, which is in a kernel lock
lkb->lkb_astparam, this field should never be derefenced by the DLM
kernel stack. In case of a user lock lkb->lkb_astparam is lkb->lkb_ua
(memory is shared by a union field). The struct lkb_ua will be handled
by the DLM kernel stack but on a kernel lock it will contain invalid
data and ends in most likely crashing the kernel.
It can be reproduced with two cluster nodes.
node 2:
dlm_tool join test
echo "862 fooobaar 1 2 1" > /sys/kernel/debug/dlm/test_locks
echo "862 3 1" > /sys/kernel/debug/dlm/test_waiters
node 1:
dlm_tool join test
python:
foo = DLM(h_cmd=3, o_nextcmd=1, h_nodeid=1, h_lockspace=0x77222027, \
m_type=7, m_flags=0x1, m_remid=0x862, m_result=0xFFFEFFFE)
newFile = open("/sys/kernel/debug/dlm/comms/2/rawmsg", "wb")
newFile.write(bytes(foo))
Signed-off-by: Alexander Aring <aahringo@redhat.com>
Signed-off-by: David Teigland <teigland@redhat.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
fs/dlm/lock.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/fs/dlm/lock.c b/fs/dlm/lock.c
index a928ba008d7d7..26a4847efccca 100644
--- a/fs/dlm/lock.c
+++ b/fs/dlm/lock.c
@@ -3977,6 +3977,14 @@ static int validate_message(struct dlm_lkb *lkb, struct dlm_message *ms)
int from = ms->m_header.h_nodeid;
int error = 0;
+ /* currently mixing of user/kernel locks are not supported */
+ if (ms->m_flags & DLM_IFL_USER && ~lkb->lkb_flags & DLM_IFL_USER) {
+ log_error(lkb->lkb_resource->res_ls,
+ "got user dlm message for a kernel lock");
+ error = -EINVAL;
+ goto out;
+ }
+
switch (ms->m_type) {
case DLM_MSG_CONVERT:
case DLM_MSG_UNLOCK:
@@ -4005,6 +4013,7 @@ static int validate_message(struct dlm_lkb *lkb, struct dlm_message *ms)
error = -EINVAL;
}
+out:
if (error)
log_error(lkb->lkb_resource->res_ls,
"ignore invalid message %d from %d %x %x %x %d",
--
2.34.1
WARNING: multiple messages have this Message-ID (diff)
From: Sasha Levin <sashal@kernel.org>
To: linux-kernel@vger.kernel.org, stable@vger.kernel.org
Cc: Alexander Aring <aahringo@redhat.com>,
David Teigland <teigland@redhat.com>,
Sasha Levin <sashal@kernel.org>,
ccaulfie@redhat.com, cluster-devel@redhat.com
Subject: [PATCH AUTOSEL 4.19 02/59] fs: dlm: filter user dlm messages for kernel locks
Date: Mon, 17 Jan 2022 21:46:03 -0500 [thread overview]
Message-ID: <20220118024701.1952911-2-sashal@kernel.org> (raw)
In-Reply-To: <20220118024701.1952911-1-sashal@kernel.org>
From: Alexander Aring <aahringo@redhat.com>
[ Upstream commit 6c2e3bf68f3e5e5a647aa52be246d5f552d7496d ]
This patch fixes the following crash by receiving a invalid message:
[ 160.672220] ==================================================================
[ 160.676206] BUG: KASAN: user-memory-access in dlm_user_add_ast+0xc3/0x370
[ 160.679659] Read of size 8 at addr 00000000deadbeef by task kworker/u32:13/319
[ 160.681447]
[ 160.681824] CPU: 10 PID: 319 Comm: kworker/u32:13 Not tainted 5.14.0-rc2+ #399
[ 160.683472] Hardware name: Red Hat KVM/RHEL-AV, BIOS 1.14.0-1.module+el8.6.0+12648+6ede71a5 04/01/2014
[ 160.685574] Workqueue: dlm_recv process_recv_sockets
[ 160.686721] Call Trace:
[ 160.687310] dump_stack_lvl+0x56/0x6f
[ 160.688169] ? dlm_user_add_ast+0xc3/0x370
[ 160.689116] kasan_report.cold.14+0x116/0x11b
[ 160.690138] ? dlm_user_add_ast+0xc3/0x370
[ 160.690832] dlm_user_add_ast+0xc3/0x370
[ 160.691502] _receive_unlock_reply+0x103/0x170
[ 160.692241] _receive_message+0x11df/0x1ec0
[ 160.692926] ? rcu_read_lock_sched_held+0xa1/0xd0
[ 160.693700] ? rcu_read_lock_bh_held+0xb0/0xb0
[ 160.694427] ? lock_acquire+0x175/0x400
[ 160.695058] ? do_purge.isra.51+0x200/0x200
[ 160.695744] ? lock_acquired+0x360/0x5d0
[ 160.696400] ? lock_contended+0x6a0/0x6a0
[ 160.697055] ? lock_release+0x21d/0x5e0
[ 160.697686] ? lock_is_held_type+0xe0/0x110
[ 160.698352] ? lock_is_held_type+0xe0/0x110
[ 160.699026] ? ___might_sleep+0x1cc/0x1e0
[ 160.699698] ? dlm_wait_requestqueue+0x94/0x140
[ 160.700451] ? dlm_process_requestqueue+0x240/0x240
[ 160.701249] ? down_write_killable+0x2b0/0x2b0
[ 160.701988] ? do_raw_spin_unlock+0xa2/0x130
[ 160.702690] dlm_receive_buffer+0x1a5/0x210
[ 160.703385] dlm_process_incoming_buffer+0x726/0x9f0
[ 160.704210] receive_from_sock+0x1c0/0x3b0
[ 160.704886] ? dlm_tcp_shutdown+0x30/0x30
[ 160.705561] ? lock_acquire+0x175/0x400
[ 160.706197] ? rcu_read_lock_sched_held+0xa1/0xd0
[ 160.706941] ? rcu_read_lock_bh_held+0xb0/0xb0
[ 160.707681] process_recv_sockets+0x32/0x40
[ 160.708366] process_one_work+0x55e/0xad0
[ 160.709045] ? pwq_dec_nr_in_flight+0x110/0x110
[ 160.709820] worker_thread+0x65/0x5e0
[ 160.710423] ? process_one_work+0xad0/0xad0
[ 160.711087] kthread+0x1ed/0x220
[ 160.711628] ? set_kthread_struct+0x80/0x80
[ 160.712314] ret_from_fork+0x22/0x30
The issue is that we received a DLM message for a user lock but the
destination lock is a kernel lock. Note that the address which is trying
to derefence is 00000000deadbeef, which is in a kernel lock
lkb->lkb_astparam, this field should never be derefenced by the DLM
kernel stack. In case of a user lock lkb->lkb_astparam is lkb->lkb_ua
(memory is shared by a union field). The struct lkb_ua will be handled
by the DLM kernel stack but on a kernel lock it will contain invalid
data and ends in most likely crashing the kernel.
It can be reproduced with two cluster nodes.
node 2:
dlm_tool join test
echo "862 fooobaar 1 2 1" > /sys/kernel/debug/dlm/test_locks
echo "862 3 1" > /sys/kernel/debug/dlm/test_waiters
node 1:
dlm_tool join test
python:
foo = DLM(h_cmd=3, o_nextcmd=1, h_nodeid=1, h_lockspace=0x77222027, \
m_type=7, m_flags=0x1, m_remid=0x862, m_result=0xFFFEFFFE)
newFile = open("/sys/kernel/debug/dlm/comms/2/rawmsg", "wb")
newFile.write(bytes(foo))
Signed-off-by: Alexander Aring <aahringo@redhat.com>
Signed-off-by: David Teigland <teigland@redhat.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
fs/dlm/lock.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/fs/dlm/lock.c b/fs/dlm/lock.c
index a928ba008d7d7..26a4847efccca 100644
--- a/fs/dlm/lock.c
+++ b/fs/dlm/lock.c
@@ -3977,6 +3977,14 @@ static int validate_message(struct dlm_lkb *lkb, struct dlm_message *ms)
int from = ms->m_header.h_nodeid;
int error = 0;
+ /* currently mixing of user/kernel locks are not supported */
+ if (ms->m_flags & DLM_IFL_USER && ~lkb->lkb_flags & DLM_IFL_USER) {
+ log_error(lkb->lkb_resource->res_ls,
+ "got user dlm message for a kernel lock");
+ error = -EINVAL;
+ goto out;
+ }
+
switch (ms->m_type) {
case DLM_MSG_CONVERT:
case DLM_MSG_UNLOCK:
@@ -4005,6 +4013,7 @@ static int validate_message(struct dlm_lkb *lkb, struct dlm_message *ms)
error = -EINVAL;
}
+out:
if (error)
log_error(lkb->lkb_resource->res_ls,
"ignore invalid message %d from %d %x %x %x %d",
--
2.34.1
next prev parent reply other threads:[~2022-01-18 2:46 UTC|newest]
Thread overview: 74+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-01-18 2:46 [PATCH AUTOSEL 4.19 01/59] Bluetooth: Fix debugfs entry leak in hci_register_dev() Sasha Levin
2022-01-18 2:46 ` Sasha Levin [this message]
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 02/59] fs: dlm: filter user dlm messages for kernel locks Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 03/59] ar5523: Fix null-ptr-deref with unexpected WDCMSG_TARGET_START reply Sasha Levin
2022-01-18 2:46 ` [Nouveau] [PATCH AUTOSEL 4.19 04/59] drm/nouveau/pmu/gm200-: avoid touching PMU outside of DEVINIT/PREOS/ACR Sasha Levin
2022-01-18 2:46 ` Sasha Levin
2022-01-18 2:46 ` Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 05/59] usb: gadget: f_fs: Use stream_open() for endpoint files Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 06/59] HID: apple: Do not reset quirks when the Fn key is not found Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 07/59] media: b2c2: Add missing check in flexcop_pci_isr: Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 08/59] ARM: imx: rename DEBUG_IMX21_IMX27_UART to DEBUG_IMX27_UART Sasha Levin
2022-01-18 2:46 ` Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 09/59] mlxsw: pci: Add shutdown method in PCI driver Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 10/59] drm/bridge: megachips: Ensure both bridges are probed before registration Sasha Levin
2022-01-18 2:46 ` Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 11/59] gpiolib: acpi: Do not set the IRQ type if the IRQ is already in use Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 12/59] HSI: core: Fix return freed object in hsi_new_client Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 13/59] mwifiex: Fix skb_over_panic in mwifiex_usb_recv() Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 14/59] rsi: Fix out-of-bounds read in rsi_read_pkt() Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 15/59] usb: uhci: add aspeed ast2600 uhci support Sasha Levin
2022-01-18 2:46 ` Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 16/59] floppy: Add max size check for user space request Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 17/59] media: uvcvideo: Increase UVC_CTRL_CONTROL_TIMEOUT to 5 seconds Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 18/59] media: saa7146: hexium_orion: Fix a NULL pointer dereference in hexium_attach() Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 19/59] media: m920x: don't use stack on USB reads Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 20/59] iwlwifi: mvm: synchronize with FW after multicast commands Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 21/59] ath10k: Fix tx hanging Sasha Levin
2022-01-18 2:46 ` Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 22/59] net-sysfs: update the queue counts in the unregistration path Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 23/59] net: Enable neighbor sysctls that is save for userns root Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 24/59] x86/mce: Mark mce_panic() noinstr Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 25/59] x86/mce: Mark mce_end() noinstr Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 26/59] x86/mce: Mark mce_read_aux() noinstr Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 27/59] net: bonding: debug: avoid printing debug logs when bond is not notifying peers Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 28/59] bpf: Do not WARN in bpf_warn_invalid_xdp_action() Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 29/59] HID: quirks: Allow inverting the absolute X/Y values Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 30/59] media: igorplugusb: receiver overflow should be reported Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 31/59] media: saa7146: hexium_gemini: Fix a NULL pointer dereference in hexium_attach() Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 32/59] mmc: core: Fixup storing of OCR for MMC_QUIRK_NONSTD_SDIO Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 33/59] audit: ensure userspace is penalized the same as the kernel when under pressure Sasha Levin
2022-01-18 2:46 ` Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 34/59] arm64: tegra: Adjust length of CCPLEX cluster MMIO region Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 35/59] usb: hub: Add delay for SuperSpeed hub resume to let links transit to U0 Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 36/59] ath9k: Fix out-of-bound memcpy in ath9k_hif_usb_rx_stream Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 37/59] iwlwifi: fix leaks/bad data after failed firmware load Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 38/59] iwlwifi: remove module loading failure message Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 39/59] iwlwifi: mvm: Fix calculation of frame length Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 40/59] um: registers: Rename function names to avoid conflicts and build problems Sasha Levin
2022-01-18 2:46 ` Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 41/59] jffs2: GC deadlock reading a page that is used in jffs2_write_begin() Sasha Levin
2022-01-18 2:46 ` Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 42/59] ACPICA: actypes.h: Expand the ACPI_ACCESS_ definitions Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 43/59] ACPICA: Utilities: Avoid deleting the same object twice in a row Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 44/59] ACPICA: Executer: Fix the REFCLASS_REFOF case in acpi_ex_opcode_1A_0T_1R() Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 45/59] ACPICA: Hardware: Do not flush CPU cache when entering S4 and S5 Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 46/59] drm/amdgpu: fixup bad vram size on gmc v8 Sasha Levin
2022-01-18 2:46 ` Sasha Levin
2022-01-18 2:46 ` Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 47/59] ACPI: battery: Add the ThinkPad "Not Charging" quirk Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 48/59] btrfs: remove BUG_ON() in find_parent_nodes() Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 49/59] btrfs: remove BUG_ON(!eie) in find_parent_nodes Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 50/59] net: mdio: Demote probed message to debug print Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 51/59] mac80211: allow non-standard VHT MCS-10/11 Sasha Levin
2022-01-18 2:46 ` [dm-devel] [PATCH AUTOSEL 4.19 52/59] dm btree: add a defensive bounds check to insert_at() Sasha Levin
2022-01-18 2:46 ` Sasha Levin
2022-01-18 2:46 ` [dm-devel] [PATCH AUTOSEL 4.19 53/59] dm space map common: add bounds check to sm_ll_lookup_bitmap() Sasha Levin
2022-01-18 2:46 ` Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 54/59] net: phy: marvell: configure RGMII delays for 88E1118 Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 55/59] net: gemini: allow any RGMII interface mode Sasha Levin
2022-01-18 2:46 ` Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 56/59] regulator: qcom_smd: Align probe function with rpmh-regulator Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 57/59] serial: pl010: Drop CR register reset on set_termios Sasha Levin
2022-01-18 2:46 ` [PATCH AUTOSEL 4.19 58/59] serial: core: Keep mctrl register state and cached copy in sync Sasha Levin
2022-01-18 2:47 ` [PATCH AUTOSEL 4.19 59/59] parisc: Avoid calling faulthandler_disabled() twice Sasha Levin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220118024701.1952911-2-sashal@kernel.org \
--to=sashal@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.