[PATCH] media: dvb-core: add upper bound check in DMX_SET_BUFFER_SIZE ioctl

[PATCH] media: dvb-core: add upper bound check in DMX_SET_BUFFER_SIZE ioctl

[Cen Zhang (Microsoft)]

dvb_dvr_set_buffer_size() and dvb_dmxdev_set_buffer_size() pass the
user-supplied size argument directly to vmalloc() without any upper
bound check. This allows excessive kernel memory allocation via the
DMX_SET_BUFFER_SIZE ioctl, which can lead to system-wide OOM conditions.

  Kernel panic - not syncing: System is deadlocked on memory

  Call Trace:
   out_of_memory+0x12fd/0x1370
   __alloc_frozen_pages_noprof+0x2620/0x2fa0
   __vmalloc_node_range_noprof+0x7fa/0x1490
   dvb_dvr_do_ioctl+0x11e/0x260 (drivers/media/dvb-core/dmxdev.c:296)
   dvb_usercopy+0x15b/0x360

Fix by cap both functions at 64 MB and return -EINVAL for oversized
requests.

Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
Fixes: a095be4b030c ("V4L/DVB (7659): dvb-core: Implement DMX_SET_BUFFER_SIZE for dvr")
Reported-by: AutonomousCodeSecurity@microsoft.com
Signed-off-by: Cen Zhang (Microsoft) <blbllhy@gmail.com>
---
 drivers/media/dvb-core/dmxdev.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/drivers/media/dvb-core/dmxdev.c b/drivers/media/dvb-core/dmxdev.c
index 3c8bc75e4d6c..b4dc87945be1 100644
--- a/drivers/media/dvb-core/dmxdev.c
+++ b/drivers/media/dvb-core/dmxdev.c
@@ -20,6 +20,9 @@
 #include <media/dmxdev.h>
 #include <media/dvb_vb2.h>
 
+/* 64 MB upper bound for DVB ring buffer allocations */
+#define DVB_BUFFER_SIZE_MAX (64 * 1024 * 1024)
+
 static int debug;
 
 module_param(debug, int, 0644);
@@ -292,6 +295,8 @@ static int dvb_dvr_set_buffer_size(struct dmxdev *dmxdev,
 		return 0;
 	if (!size)
 		return -EINVAL;
+	if (size > DVB_BUFFER_SIZE_MAX)
+		return -EINVAL;
 
 	newmem = vmalloc(size);
 	if (!newmem)
@@ -333,6 +338,8 @@ static int dvb_dmxdev_set_buffer_size(struct dmxdev_filter *dmxdevfilter,
 		return -EINVAL;
 	if (dmxdevfilter->state >= DMXDEV_STATE_GO)
 		return -EBUSY;
+	if (size > DVB_BUFFER_SIZE_MAX)
+		return -EINVAL;
 
 	newmem = vmalloc(size);
 	if (!newmem)
-- 
2.53.0

Re: [PATCH] media: dvb-core: add upper bound check in DMX_SET_BUFFER_SIZE ioctl

[Mauro Carvalho Chehab]

On Wed, 22 Jul 2026 15:37:19 -0400
"Cen Zhang (Microsoft)" <blbllhy@gmail.com> wrote:

> dvb_dvr_set_buffer_size() and dvb_dmxdev_set_buffer_size() pass the
> user-supplied size argument directly to vmalloc() without any upper
> bound check. This allows excessive kernel memory allocation via the
> DMX_SET_BUFFER_SIZE ioctl, which can lead to system-wide OOM conditions.
> 
>   Kernel panic - not syncing: System is deadlocked on memory
> 
>   Call Trace:
>    out_of_memory+0x12fd/0x1370
>    __alloc_frozen_pages_noprof+0x2620/0x2fa0
>    __vmalloc_node_range_noprof+0x7fa/0x1490
>    dvb_dvr_do_ioctl+0x11e/0x260 (drivers/media/dvb-core/dmxdev.c:296)
>    dvb_usercopy+0x15b/0x360
> 
> Fix by cap both functions at 64 MB and return -EINVAL for oversized
> requests.
> 
> Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
> Fixes: a095be4b030c ("V4L/DVB (7659): dvb-core: Implement DMX_SET_BUFFER_SIZE for dvr")
> Reported-by: AutonomousCodeSecurity@microsoft.com
> Signed-off-by: Cen Zhang (Microsoft) <blbllhy@gmail.com>
> ---
>  drivers/media/dvb-core/dmxdev.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/drivers/media/dvb-core/dmxdev.c b/drivers/media/dvb-core/dmxdev.c
> index 3c8bc75e4d6c..b4dc87945be1 100644
> --- a/drivers/media/dvb-core/dmxdev.c
> +++ b/drivers/media/dvb-core/dmxdev.c
> @@ -20,6 +20,9 @@
>  #include <media/dmxdev.h>
>  #include <media/dvb_vb2.h>
>  
> +/* 64 MB upper bound for DVB ring buffer allocations */
> +#define DVB_BUFFER_SIZE_MAX (64 * 1024 * 1024)

Having a limit is good, but why 64MB?

Can you provide me more details about the test scenario: e.g. with
what TV standards had you test it, and such.

Regards,
Mauro

Thanks,
Mauro

Re: [PATCH] media: dvb-core: add upper bound check in DMX_SET_BUFFER_SIZE ioctl

[Mauro Carvalho Chehab]

On Wed, 29 Jul 2026 08:30:26 +0200
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:

> On Wed, 22 Jul 2026 15:37:19 -0400
> "Cen Zhang (Microsoft)" <blbllhy@gmail.com> wrote:
> 
> > dvb_dvr_set_buffer_size() and dvb_dmxdev_set_buffer_size() pass the
> > user-supplied size argument directly to vmalloc() without any upper
> > bound check. This allows excessive kernel memory allocation via the
> > DMX_SET_BUFFER_SIZE ioctl, which can lead to system-wide OOM conditions.
> > 
> >   Kernel panic - not syncing: System is deadlocked on memory
> > 
> >   Call Trace:
> >    out_of_memory+0x12fd/0x1370
> >    __alloc_frozen_pages_noprof+0x2620/0x2fa0
> >    __vmalloc_node_range_noprof+0x7fa/0x1490
> >    dvb_dvr_do_ioctl+0x11e/0x260 (drivers/media/dvb-core/dmxdev.c:296)
> >    dvb_usercopy+0x15b/0x360
> > 
> > Fix by cap both functions at 64 MB and return -EINVAL for oversized
> > requests.
> > 
> > Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
> > Fixes: a095be4b030c ("V4L/DVB (7659): dvb-core: Implement DMX_SET_BUFFER_SIZE for dvr")
> > Reported-by: AutonomousCodeSecurity@microsoft.com
> > Signed-off-by: Cen Zhang (Microsoft) <blbllhy@gmail.com>
> > ---
> >  drivers/media/dvb-core/dmxdev.c | 7 +++++++
> >  1 file changed, 7 insertions(+)
> > 
> > diff --git a/drivers/media/dvb-core/dmxdev.c b/drivers/media/dvb-core/dmxdev.c
> > index 3c8bc75e4d6c..b4dc87945be1 100644
> > --- a/drivers/media/dvb-core/dmxdev.c
> > +++ b/drivers/media/dvb-core/dmxdev.c
> > @@ -20,6 +20,9 @@
> >  #include <media/dmxdev.h>
> >  #include <media/dvb_vb2.h>
> >  
> > +/* 64 MB upper bound for DVB ring buffer allocations */
> > +#define DVB_BUFFER_SIZE_MAX (64 * 1024 * 1024)
> 
> Having a limit is good, but why 64MB?
> 
> Can you provide me more details about the test scenario: e.g. with
> what TV standards had you test it, and such.

Forgot to mention, but instead of returning -EINVAL, probably the best
would be to setup the ringbuffer size to the maximum value, as
otherwise this would break existing apps.

Btw, what apps did you use to test it? Had you check what's the current
limit on what apps?

Regards,
Mauro

Re: [PATCH] media: dvb-core: add upper bound check in DMX_SET_BUFFER_SIZE ioctl

[Cen Zhang (Microsoft)]

Hi Mauro,

> Forgot to mention, but instead of returning -EINVAL, probably the best
> would be to setup the ringbuffer size to the maximum value, as
> otherwise this would break existing apps.

Thanks for pointing this out. I will revise this in v3.

> Having a limit is good, but why 64MB?
>
> Btw, what apps did you use to test it? Had you check what's the current
> limit on what apps?

I chose 64 MiB as a conservative upper bound based on a brief source
survey of DVB applications:

- The kernel DVR buffer defaults to approximately 1.84 MiB. In the
  upstream revisions I checked, TVheadend, MythTV and MuMuDVB do not
  actively call DMX_SET_BUFFER_SIZE and therefore retain this default:

https://github.com/torvalds/linux/blob/acb7500801e98639f6d8c2d796ed9f64cba83d3a/include/media/dmxdev.h#L189

- SATPI allows approximately 30 MiB:

https://github.com/Barracuda09/SATPI/blob/09df7402870b1dff2c7b2d7f4a085002a333045f/src/input/dvb/Frontend.cpp#L57

- DVBlast defaults to approximately 7.34 MiB:

https://github.com/videolan/dvblast/blob/ae6b24ae8cdf6eaa1ada8534a257258f3c906a77/dvb.c#L71

- dvbv5-zap uses approximately 5.88 MiB:

https://github.com/gjasny/v4l-utils/blob/95ad25f6a77a0a6650f5f657ac2c5046efcd04a0/utils/dvb/dvbv5-zap.c#L32

- minisatip defaults to approximately 5.51 MiB:

https://github.com/catalinii/minisatip/blob/d20136e23d47bd7f5de2967b34463f14bce6f988/src/adapter.h#L12

I have limited experience with DVB hardware and production workloads.
Do you have a recommended limit based on known hardware or application
requirements? I am happy to use the value you recommend in v3.

> Can you provide me more details about the test scenario: e.g. with
> what TV standards had you test it, and such.

The reproducer used the vidtv virtual DVB adapter
(dvb-vidtv-bridge) in a QEMU VM with approximately 3 GiB of RAM. The
root-only setup loaded the driver, made dvr0 accessible and set
oom_score_adj=-1000 for the process lineage to make the panic
deterministic. The ioctl itself was issued by an unprivileged process.

The test did not exercise a particular TV standard or transport stream,
since the failure occurs directly in the buffer-allocation ioctl path
before stream processing. I have not tested this on physical DVB
hardware.

I can share the reproducer package privately if that would be useful.

Best regards,
Cen