[PATCH] media: dvb-core: add upper bound check in DMX_SET_BUFFER_SIZE ioctl

[PATCH] media: dvb-core: add upper bound check in DMX_SET_BUFFER_SIZE ioctl

[Cen Zhang (Microsoft)]

dvb_dvr_set_buffer_size() and dvb_dmxdev_set_buffer_size() pass the
user-supplied size argument directly to vmalloc() without any upper
bound check. This allows excessive kernel memory allocation via the
DMX_SET_BUFFER_SIZE ioctl, which can lead to system-wide OOM conditions.

  Kernel panic - not syncing: System is deadlocked on memory

  Call Trace:
   out_of_memory+0x12fd/0x1370
   __alloc_frozen_pages_noprof+0x2620/0x2fa0
   __vmalloc_node_range_noprof+0x7fa/0x1490
   dvb_dvr_do_ioctl+0x11e/0x260 (drivers/media/dvb-core/dmxdev.c:296)
   dvb_usercopy+0x15b/0x360

Fix by cap both functions at 64 MB and return -EINVAL for oversized
requests.

Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
Fixes: a095be4b030c ("V4L/DVB (7659): dvb-core: Implement DMX_SET_BUFFER_SIZE for dvr")
Reported-by: AutonomousCodeSecurity@microsoft.com
Signed-off-by: Cen Zhang (Microsoft) <blbllhy@gmail.com>
---
 drivers/media/dvb-core/dmxdev.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/drivers/media/dvb-core/dmxdev.c b/drivers/media/dvb-core/dmxdev.c
index 3c8bc75e4d6c..b4dc87945be1 100644
--- a/drivers/media/dvb-core/dmxdev.c
+++ b/drivers/media/dvb-core/dmxdev.c
@@ -20,6 +20,9 @@
 #include <media/dmxdev.h>
 #include <media/dvb_vb2.h>
 
+/* 64 MB upper bound for DVB ring buffer allocations */
+#define DVB_BUFFER_SIZE_MAX (64 * 1024 * 1024)
+
 static int debug;
 
 module_param(debug, int, 0644);
@@ -292,6 +295,8 @@ static int dvb_dvr_set_buffer_size(struct dmxdev *dmxdev,
 		return 0;
 	if (!size)
 		return -EINVAL;
+	if (size > DVB_BUFFER_SIZE_MAX)
+		return -EINVAL;
 
 	newmem = vmalloc(size);
 	if (!newmem)
@@ -333,6 +338,8 @@ static int dvb_dmxdev_set_buffer_size(struct dmxdev_filter *dmxdevfilter,
 		return -EINVAL;
 	if (dmxdevfilter->state >= DMXDEV_STATE_GO)
 		return -EBUSY;
+	if (size > DVB_BUFFER_SIZE_MAX)
+		return -EINVAL;
 
 	newmem = vmalloc(size);
 	if (!newmem)
-- 
2.53.0

Re: [PATCH] media: dvb-core: add upper bound check in DMX_SET_BUFFER_SIZE ioctl

[Mauro Carvalho Chehab]

On Wed, 22 Jul 2026 15:37:19 -0400
"Cen Zhang (Microsoft)" <blbllhy@gmail.com> wrote:

> dvb_dvr_set_buffer_size() and dvb_dmxdev_set_buffer_size() pass the
> user-supplied size argument directly to vmalloc() without any upper
> bound check. This allows excessive kernel memory allocation via the
> DMX_SET_BUFFER_SIZE ioctl, which can lead to system-wide OOM conditions.
> 
>   Kernel panic - not syncing: System is deadlocked on memory
> 
>   Call Trace:
>    out_of_memory+0x12fd/0x1370
>    __alloc_frozen_pages_noprof+0x2620/0x2fa0
>    __vmalloc_node_range_noprof+0x7fa/0x1490
>    dvb_dvr_do_ioctl+0x11e/0x260 (drivers/media/dvb-core/dmxdev.c:296)
>    dvb_usercopy+0x15b/0x360
> 
> Fix by cap both functions at 64 MB and return -EINVAL for oversized
> requests.
> 
> Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
> Fixes: a095be4b030c ("V4L/DVB (7659): dvb-core: Implement DMX_SET_BUFFER_SIZE for dvr")
> Reported-by: AutonomousCodeSecurity@microsoft.com
> Signed-off-by: Cen Zhang (Microsoft) <blbllhy@gmail.com>
> ---
>  drivers/media/dvb-core/dmxdev.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/drivers/media/dvb-core/dmxdev.c b/drivers/media/dvb-core/dmxdev.c
> index 3c8bc75e4d6c..b4dc87945be1 100644
> --- a/drivers/media/dvb-core/dmxdev.c
> +++ b/drivers/media/dvb-core/dmxdev.c
> @@ -20,6 +20,9 @@
>  #include <media/dmxdev.h>
>  #include <media/dvb_vb2.h>
>  
> +/* 64 MB upper bound for DVB ring buffer allocations */
> +#define DVB_BUFFER_SIZE_MAX (64 * 1024 * 1024)

Having a limit is good, but why 64MB?

Can you provide me more details about the test scenario: e.g. with
what TV standards had you test it, and such.

Regards,
Mauro

Thanks,
Mauro

Re: [PATCH] media: dvb-core: add upper bound check in DMX_SET_BUFFER_SIZE ioctl

[Mauro Carvalho Chehab]

On Wed, 29 Jul 2026 08:30:26 +0200
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:

> On Wed, 22 Jul 2026 15:37:19 -0400
> "Cen Zhang (Microsoft)" <blbllhy@gmail.com> wrote:
> 
> > dvb_dvr_set_buffer_size() and dvb_dmxdev_set_buffer_size() pass the
> > user-supplied size argument directly to vmalloc() without any upper
> > bound check. This allows excessive kernel memory allocation via the
> > DMX_SET_BUFFER_SIZE ioctl, which can lead to system-wide OOM conditions.
> > 
> >   Kernel panic - not syncing: System is deadlocked on memory
> > 
> >   Call Trace:
> >    out_of_memory+0x12fd/0x1370
> >    __alloc_frozen_pages_noprof+0x2620/0x2fa0
> >    __vmalloc_node_range_noprof+0x7fa/0x1490
> >    dvb_dvr_do_ioctl+0x11e/0x260 (drivers/media/dvb-core/dmxdev.c:296)
> >    dvb_usercopy+0x15b/0x360
> > 
> > Fix by cap both functions at 64 MB and return -EINVAL for oversized
> > requests.
> > 
> > Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
> > Fixes: a095be4b030c ("V4L/DVB (7659): dvb-core: Implement DMX_SET_BUFFER_SIZE for dvr")
> > Reported-by: AutonomousCodeSecurity@microsoft.com
> > Signed-off-by: Cen Zhang (Microsoft) <blbllhy@gmail.com>
> > ---
> >  drivers/media/dvb-core/dmxdev.c | 7 +++++++
> >  1 file changed, 7 insertions(+)
> > 
> > diff --git a/drivers/media/dvb-core/dmxdev.c b/drivers/media/dvb-core/dmxdev.c
> > index 3c8bc75e4d6c..b4dc87945be1 100644
> > --- a/drivers/media/dvb-core/dmxdev.c
> > +++ b/drivers/media/dvb-core/dmxdev.c
> > @@ -20,6 +20,9 @@
> >  #include <media/dmxdev.h>
> >  #include <media/dvb_vb2.h>
> >  
> > +/* 64 MB upper bound for DVB ring buffer allocations */
> > +#define DVB_BUFFER_SIZE_MAX (64 * 1024 * 1024)
> 
> Having a limit is good, but why 64MB?
> 
> Can you provide me more details about the test scenario: e.g. with
> what TV standards had you test it, and such.

Forgot to mention, but instead of returning -EINVAL, probably the best
would be to setup the ringbuffer size to the maximum value, as
otherwise this would break existing apps.

Btw, what apps did you use to test it? Had you check what's the current
limit on what apps?

Regards,
Mauro