[PATCH v3] media: dvb-core: add upper bound check in DMX_SET_BUFFER_SIZE ioctl

["Cen Zhang (Microsoft)" <blbllhy@gmail.com>]

dvb_dvr_set_buffer_size() and dvb_dmxdev_set_buffer_size() pass the
user-supplied size argument directly to vmalloc() without any upper
bound check. This allows excessive kernel memory allocation via the
DMX_SET_BUFFER_SIZE ioctl, which can lead to system-wide OOM
conditions.

  Kernel panic - not syncing: System is deadlocked on memory

  Call Trace:
   out_of_memory+0x12fd/0x1370
   __alloc_frozen_pages_noprof+0x2620/0x2fa0
   __vmalloc_node_range_noprof+0x7fa/0x1490
   dvb_dvr_do_ioctl+0x11e/0x260 (drivers/media/dvb-core/dmxdev.c:296)
   dvb_usercopy+0x15b/0x360

Fix by clamping both functions to 64 MiB for oversized requests. A
quick survey of SATPI, DVBlast, dvbv5-zap, minisatip, TVheadend,
MythTV, and MuMuDVB found the largest requested buffer to be about
30 MiB.

Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
Fixes: a095be4b030c ("V4L/DVB (7659): dvb-core: Implement DMX_SET_BUFFER_SIZE for dvr")
Cc: stable@vger.kernel.org
Reported-by: AutonomousCodeSecurity@microsoft.com
Link: https://lore.kernel.org/all/20260722193719.81157-1-blbllhy@gmail.com/
Signed-off-by: Cen Zhang (Microsoft) <blbllhy@gmail.com>
---
V3:
 - Clamp oversized requests to maximum instead of returning -EINVAL.
 - Clamp before size comparison to avoid redundant reallocation.
 - Explain the choice of the 64 MiB limit.

V2:
 - Add Cc: stable and Link: tag.

 drivers/media/dvb-core/dmxdev.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/drivers/media/dvb-core/dmxdev.c b/drivers/media/dvb-core/dmxdev.c
index 3c8bc75e4d6c..330255312c85 100644
--- a/drivers/media/dvb-core/dmxdev.c
+++ b/drivers/media/dvb-core/dmxdev.c
@@ -20,6 +20,9 @@
 #include <media/dmxdev.h>
 #include <media/dvb_vb2.h>
 
+/* 64 MiB upper bound for DVB ring buffer allocations */
+#define DVB_BUFFER_SIZE_MAX (64 * 1024 * 1024)
+
 static int debug;
 
 module_param(debug, int, 0644);
@@ -288,6 +291,8 @@ static int dvb_dvr_set_buffer_size(struct dmxdev *dmxdev,
 
 	dprintk("%s\n", __func__);
 
+	if (size > DVB_BUFFER_SIZE_MAX)
+		size = DVB_BUFFER_SIZE_MAX;
 	if (buf->size == size)
 		return 0;
 	if (!size)
@@ -327,6 +332,8 @@ static int dvb_dmxdev_set_buffer_size(struct dmxdev_filter *dmxdevfilter,
 	void *newmem;
 	void *oldmem;
 
+	if (size > DVB_BUFFER_SIZE_MAX)
+		size = DVB_BUFFER_SIZE_MAX;
 	if (buf->size == size)
 		return 0;
 	if (!size)
-- 
2.52.0