* Re: [PATCH v3 04/11] media: videobuf2: Stop define VB2_MAX_FRAME as global [not found] ` <20230622131349.144160-5-benjamin.gaignard@collabora.com> @ 2023-06-30 9:51 ` Hsia-Jun Li 2023-07-03 8:09 ` Benjamin Gaignard 0 siblings, 1 reply; 27+ messages in thread From: Hsia-Jun Li @ 2023-06-30 9:51 UTC (permalink / raw) To: Benjamin Gaignard Cc: linux-media, linux-kernel, linux-arm-kernel, m.szyprowski, linux-mediatek, linux-arm-msm, hverkuil-cisco, ezequiel, p.zabel, linux-rockchip, mchehab, linux-staging, ming.qian, kernel, gregkh, tfiga, nicolas.dufresne On 6/22/23 21:13, Benjamin Gaignard wrote: > CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe. > > > After changing bufs arrays to a dynamic allocated array > VB2_MAX_FRAME doesn't mean anything for videobuf2 core. I think make it 64 which is the VB2_MAX_FRAME in Android GKI kernel is more reasonable. It would be hard to iterate the whole array, it would go worse with a filter. Such iterate may need to go twice because you mix post-processing buffer and decoding buffer(with MV) in the same array. > Remove it from the core definitions but keep it for drivers internal > needs. > > Signed-off-by: Benjamin Gaignard <benjamin.gaignard@collabora.com> > --- > drivers/media/common/videobuf2/videobuf2-core.c | 2 ++ > drivers/media/platform/amphion/vdec.c | 1 + > .../media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c | 2 ++ > drivers/media/platform/qcom/venus/hfi.h | 2 ++ > drivers/media/platform/verisilicon/hantro_hw.h | 2 ++ > drivers/staging/media/ipu3/ipu3-v4l2.c | 2 ++ > include/media/videobuf2-core.h | 1 - > include/media/videobuf2-v4l2.h | 4 ---- > 8 files changed, 11 insertions(+), 5 deletions(-) > > diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c > index 86e1e926fa45..899783f67580 100644 > --- a/drivers/media/common/videobuf2/videobuf2-core.c > +++ b/drivers/media/common/videobuf2/videobuf2-core.c > @@ -31,6 +31,8 @@ > > #include <trace/events/vb2.h> > > +#define VB2_MAX_FRAME 32 > + > static int debug; > module_param(debug, int, 0644); > > diff --git a/drivers/media/platform/amphion/vdec.c b/drivers/media/platform/amphion/vdec.c > index 3fa1a74a2e20..b3219f6d17fa 100644 > --- a/drivers/media/platform/amphion/vdec.c > +++ b/drivers/media/platform/amphion/vdec.c > @@ -28,6 +28,7 @@ > > #define VDEC_MIN_BUFFER_CAP 8 > #define VDEC_MIN_BUFFER_OUT 8 > +#define VB2_MAX_FRAME 32 > > struct vdec_fs_info { > char name[8]; > diff --git a/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c b/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c > index 6532a69f1fa8..a1e0f24bb91c 100644 > --- a/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c > +++ b/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c > @@ -16,6 +16,8 @@ > #include "../vdec_drv_if.h" > #include "../vdec_vpu_if.h" > > +#define VB2_MAX_FRAME 32 > + > /* reset_frame_context defined in VP9 spec */ > #define VP9_RESET_FRAME_CONTEXT_NONE0 0 > #define VP9_RESET_FRAME_CONTEXT_NONE1 1 > diff --git a/drivers/media/platform/qcom/venus/hfi.h b/drivers/media/platform/qcom/venus/hfi.h > index f25d412d6553..bd5ca5a8b945 100644 > --- a/drivers/media/platform/qcom/venus/hfi.h > +++ b/drivers/media/platform/qcom/venus/hfi.h > @@ -10,6 +10,8 @@ > > #include "hfi_helper.h" > > +#define VB2_MAX_FRAME 32 > + > #define VIDC_SESSION_TYPE_VPE 0 > #define VIDC_SESSION_TYPE_ENC 1 > #define VIDC_SESSION_TYPE_DEC 2 > diff --git a/drivers/media/platform/verisilicon/hantro_hw.h b/drivers/media/platform/verisilicon/hantro_hw.h > index e83f0c523a30..9e8faf7ba6fb 100644 > --- a/drivers/media/platform/verisilicon/hantro_hw.h > +++ b/drivers/media/platform/verisilicon/hantro_hw.h > @@ -15,6 +15,8 @@ > #include <media/v4l2-vp9.h> > #include <media/videobuf2-core.h> > > +#define VB2_MAX_FRAME 32 > + > #define DEC_8190_ALIGN_MASK 0x07U > > #define MB_DIM 16 > diff --git a/drivers/staging/media/ipu3/ipu3-v4l2.c b/drivers/staging/media/ipu3/ipu3-v4l2.c > index e530767e80a5..6627b5c2d4d6 100644 > --- a/drivers/staging/media/ipu3/ipu3-v4l2.c > +++ b/drivers/staging/media/ipu3/ipu3-v4l2.c > @@ -10,6 +10,8 @@ > #include "ipu3.h" > #include "ipu3-dmamap.h" > > +#define VB2_MAX_FRAME 32 > + > /******************** v4l2_subdev_ops ********************/ > > #define IPU3_RUNNING_MODE_VIDEO 0 > diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h > index 77921cf894ef..080b783d608d 100644 > --- a/include/media/videobuf2-core.h > +++ b/include/media/videobuf2-core.h > @@ -20,7 +20,6 @@ > #include <media/media-request.h> > #include <media/frame_vector.h> > > -#define VB2_MAX_FRAME (32) > #define VB2_MAX_PLANES (8) > > /** > diff --git a/include/media/videobuf2-v4l2.h b/include/media/videobuf2-v4l2.h > index 5a845887850b..88a7a565170e 100644 > --- a/include/media/videobuf2-v4l2.h > +++ b/include/media/videobuf2-v4l2.h > @@ -15,10 +15,6 @@ > #include <linux/videodev2.h> > #include <media/videobuf2-core.h> > > -#if VB2_MAX_FRAME != VIDEO_MAX_FRAME > -#error VB2_MAX_FRAME != VIDEO_MAX_FRAME > -#endif > - > #if VB2_MAX_PLANES != VIDEO_MAX_PLANES > #error VB2_MAX_PLANES != VIDEO_MAX_PLANES > #endif > -- > 2.39.2 > -- Hsia-Jun(Randy) Li ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v3 04/11] media: videobuf2: Stop define VB2_MAX_FRAME as global 2023-06-30 9:51 ` [PATCH v3 04/11] media: videobuf2: Stop define VB2_MAX_FRAME as global Hsia-Jun Li @ 2023-07-03 8:09 ` Benjamin Gaignard 2023-07-03 8:35 ` Hsia-Jun Li 0 siblings, 1 reply; 27+ messages in thread From: Benjamin Gaignard @ 2023-07-03 8:09 UTC (permalink / raw) To: Hsia-Jun Li Cc: linux-media, linux-kernel, linux-arm-kernel, m.szyprowski, linux-mediatek, linux-arm-msm, hverkuil-cisco, ezequiel, p.zabel, linux-rockchip, mchehab, linux-staging, ming.qian, kernel, gregkh, tfiga, nicolas.dufresne Le 30/06/2023 à 11:51, Hsia-Jun Li a écrit : > > On 6/22/23 21:13, Benjamin Gaignard wrote: >> CAUTION: Email originated externally, do not click links or open >> attachments unless you recognize the sender and know the content is >> safe. >> >> >> After changing bufs arrays to a dynamic allocated array >> VB2_MAX_FRAME doesn't mean anything for videobuf2 core. > > I think make it 64 which is the VB2_MAX_FRAME in Android GKI kernel is > more reasonable. > > It would be hard to iterate the whole array, it would go worse with a > filter. Such iterate may need to go twice because you mix > post-processing buffer and decoding buffer(with MV) in the same array. Here I don't want to change drivers behavior so I keep the same value. If it happens that they need more buffers, like for dynamic resolution change feature for Verisilicon VP9 decoder, case by case patches will be needed. > >> Remove it from the core definitions but keep it for drivers internal >> needs. >> >> Signed-off-by: Benjamin Gaignard <benjamin.gaignard@collabora.com> >> --- >> drivers/media/common/videobuf2/videobuf2-core.c | 2 ++ >> drivers/media/platform/amphion/vdec.c | 1 + >> .../media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c | 2 ++ >> drivers/media/platform/qcom/venus/hfi.h | 2 ++ >> drivers/media/platform/verisilicon/hantro_hw.h | 2 ++ >> drivers/staging/media/ipu3/ipu3-v4l2.c | 2 ++ >> include/media/videobuf2-core.h | 1 - >> include/media/videobuf2-v4l2.h | 4 ---- >> 8 files changed, 11 insertions(+), 5 deletions(-) >> >> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c >> b/drivers/media/common/videobuf2/videobuf2-core.c >> index 86e1e926fa45..899783f67580 100644 >> --- a/drivers/media/common/videobuf2/videobuf2-core.c >> +++ b/drivers/media/common/videobuf2/videobuf2-core.c >> @@ -31,6 +31,8 @@ >> >> #include <trace/events/vb2.h> >> >> +#define VB2_MAX_FRAME 32 >> + >> static int debug; >> module_param(debug, int, 0644); >> >> diff --git a/drivers/media/platform/amphion/vdec.c >> b/drivers/media/platform/amphion/vdec.c >> index 3fa1a74a2e20..b3219f6d17fa 100644 >> --- a/drivers/media/platform/amphion/vdec.c >> +++ b/drivers/media/platform/amphion/vdec.c >> @@ -28,6 +28,7 @@ >> >> #define VDEC_MIN_BUFFER_CAP 8 >> #define VDEC_MIN_BUFFER_OUT 8 >> +#define VB2_MAX_FRAME 32 >> >> struct vdec_fs_info { >> char name[8]; >> diff --git >> a/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >> b/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >> index 6532a69f1fa8..a1e0f24bb91c 100644 >> --- a/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >> +++ b/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >> @@ -16,6 +16,8 @@ >> #include "../vdec_drv_if.h" >> #include "../vdec_vpu_if.h" >> >> +#define VB2_MAX_FRAME 32 >> + >> /* reset_frame_context defined in VP9 spec */ >> #define VP9_RESET_FRAME_CONTEXT_NONE0 0 >> #define VP9_RESET_FRAME_CONTEXT_NONE1 1 >> diff --git a/drivers/media/platform/qcom/venus/hfi.h >> b/drivers/media/platform/qcom/venus/hfi.h >> index f25d412d6553..bd5ca5a8b945 100644 >> --- a/drivers/media/platform/qcom/venus/hfi.h >> +++ b/drivers/media/platform/qcom/venus/hfi.h >> @@ -10,6 +10,8 @@ >> >> #include "hfi_helper.h" >> >> +#define VB2_MAX_FRAME 32 >> + >> #define VIDC_SESSION_TYPE_VPE 0 >> #define VIDC_SESSION_TYPE_ENC 1 >> #define VIDC_SESSION_TYPE_DEC 2 >> diff --git a/drivers/media/platform/verisilicon/hantro_hw.h >> b/drivers/media/platform/verisilicon/hantro_hw.h >> index e83f0c523a30..9e8faf7ba6fb 100644 >> --- a/drivers/media/platform/verisilicon/hantro_hw.h >> +++ b/drivers/media/platform/verisilicon/hantro_hw.h >> @@ -15,6 +15,8 @@ >> #include <media/v4l2-vp9.h> >> #include <media/videobuf2-core.h> >> >> +#define VB2_MAX_FRAME 32 >> + >> #define DEC_8190_ALIGN_MASK 0x07U >> >> #define MB_DIM 16 >> diff --git a/drivers/staging/media/ipu3/ipu3-v4l2.c >> b/drivers/staging/media/ipu3/ipu3-v4l2.c >> index e530767e80a5..6627b5c2d4d6 100644 >> --- a/drivers/staging/media/ipu3/ipu3-v4l2.c >> +++ b/drivers/staging/media/ipu3/ipu3-v4l2.c >> @@ -10,6 +10,8 @@ >> #include "ipu3.h" >> #include "ipu3-dmamap.h" >> >> +#define VB2_MAX_FRAME 32 >> + >> /******************** v4l2_subdev_ops ********************/ >> >> #define IPU3_RUNNING_MODE_VIDEO 0 >> diff --git a/include/media/videobuf2-core.h >> b/include/media/videobuf2-core.h >> index 77921cf894ef..080b783d608d 100644 >> --- a/include/media/videobuf2-core.h >> +++ b/include/media/videobuf2-core.h >> @@ -20,7 +20,6 @@ >> #include <media/media-request.h> >> #include <media/frame_vector.h> >> >> -#define VB2_MAX_FRAME (32) >> #define VB2_MAX_PLANES (8) >> >> /** >> diff --git a/include/media/videobuf2-v4l2.h >> b/include/media/videobuf2-v4l2.h >> index 5a845887850b..88a7a565170e 100644 >> --- a/include/media/videobuf2-v4l2.h >> +++ b/include/media/videobuf2-v4l2.h >> @@ -15,10 +15,6 @@ >> #include <linux/videodev2.h> >> #include <media/videobuf2-core.h> >> >> -#if VB2_MAX_FRAME != VIDEO_MAX_FRAME >> -#error VB2_MAX_FRAME != VIDEO_MAX_FRAME >> -#endif >> - >> #if VB2_MAX_PLANES != VIDEO_MAX_PLANES >> #error VB2_MAX_PLANES != VIDEO_MAX_PLANES >> #endif >> -- >> 2.39.2 >> ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v3 04/11] media: videobuf2: Stop define VB2_MAX_FRAME as global 2023-07-03 8:09 ` Benjamin Gaignard @ 2023-07-03 8:35 ` Hsia-Jun Li 2023-07-03 10:53 ` Benjamin Gaignard 2023-07-12 10:48 ` Tomasz Figa 0 siblings, 2 replies; 27+ messages in thread From: Hsia-Jun Li @ 2023-07-03 8:35 UTC (permalink / raw) To: Benjamin Gaignard Cc: linux-media, linux-kernel, linux-arm-kernel, m.szyprowski, linux-mediatek, linux-arm-msm, hverkuil-cisco, ezequiel, p.zabel, linux-rockchip, mchehab, linux-staging, ming.qian, kernel, gregkh, tfiga, nicolas.dufresne On 7/3/23 16:09, Benjamin Gaignard wrote: > CAUTION: Email originated externally, do not click links or open > attachments unless you recognize the sender and know the content is safe. > > > Le 30/06/2023 à 11:51, Hsia-Jun Li a écrit : >> >> On 6/22/23 21:13, Benjamin Gaignard wrote: >>> CAUTION: Email originated externally, do not click links or open >>> attachments unless you recognize the sender and know the content is >>> safe. >>> >>> >>> After changing bufs arrays to a dynamic allocated array >>> VB2_MAX_FRAME doesn't mean anything for videobuf2 core. >> >> I think make it 64 which is the VB2_MAX_FRAME in Android GKI kernel is >> more reasonable. >> >> It would be hard to iterate the whole array, it would go worse with a >> filter. Such iterate may need to go twice because you mix >> post-processing buffer and decoding buffer(with MV) in the same array. > > Here I don't want to change drivers behavior so I keep the same value. > If it happens that they need more buffers, like for dynamic resolution > change > feature for Verisilicon VP9 decoder, case by case patches will be needed. > I just don't like the idea that using a variant length array here. And I could explain why you won't need so many buffers for the performance of decoding. VP9 could support 10 reference frames in dpb. Even for those frequent resolution changing test set, it would only happen to two resolutions, 32 would be enough for 20 buffers of two resolution plus golden frames. It also leaves enough slots for re-order latency. If your case had more two resolutions, likes low->medium->high. I would suggest just skip the medium resolutions, just allocate the lower one first for fast playback then the highest for all the possible medium cases. Reallocation happens frequently would only cause memory fragment, nothing benefits your performance. > >> >>> Remove it from the core definitions but keep it for drivers internal >>> needs. >>> >>> Signed-off-by: Benjamin Gaignard <benjamin.gaignard@collabora.com> >>> --- >>> drivers/media/common/videobuf2/videobuf2-core.c | 2 ++ >>> drivers/media/platform/amphion/vdec.c | 1 + >>> .../media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c | 2 ++ >>> drivers/media/platform/qcom/venus/hfi.h | 2 ++ >>> drivers/media/platform/verisilicon/hantro_hw.h | 2 ++ >>> drivers/staging/media/ipu3/ipu3-v4l2.c | 2 ++ >>> include/media/videobuf2-core.h | 1 - >>> include/media/videobuf2-v4l2.h | 4 ---- >>> 8 files changed, 11 insertions(+), 5 deletions(-) >>> >>> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c >>> b/drivers/media/common/videobuf2/videobuf2-core.c >>> index 86e1e926fa45..899783f67580 100644 >>> --- a/drivers/media/common/videobuf2/videobuf2-core.c >>> +++ b/drivers/media/common/videobuf2/videobuf2-core.c >>> @@ -31,6 +31,8 @@ >>> >>> #include <trace/events/vb2.h> >>> >>> +#define VB2_MAX_FRAME 32 >>> + >>> static int debug; >>> module_param(debug, int, 0644); >>> >>> diff --git a/drivers/media/platform/amphion/vdec.c >>> b/drivers/media/platform/amphion/vdec.c >>> index 3fa1a74a2e20..b3219f6d17fa 100644 >>> --- a/drivers/media/platform/amphion/vdec.c >>> +++ b/drivers/media/platform/amphion/vdec.c >>> @@ -28,6 +28,7 @@ >>> >>> #define VDEC_MIN_BUFFER_CAP 8 >>> #define VDEC_MIN_BUFFER_OUT 8 >>> +#define VB2_MAX_FRAME 32 >>> >>> struct vdec_fs_info { >>> char name[8]; >>> diff --git >>> a/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >>> b/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >>> index 6532a69f1fa8..a1e0f24bb91c 100644 >>> --- a/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >>> +++ b/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >>> @@ -16,6 +16,8 @@ >>> #include "../vdec_drv_if.h" >>> #include "../vdec_vpu_if.h" >>> >>> +#define VB2_MAX_FRAME 32 >>> + >>> /* reset_frame_context defined in VP9 spec */ >>> #define VP9_RESET_FRAME_CONTEXT_NONE0 0 >>> #define VP9_RESET_FRAME_CONTEXT_NONE1 1 >>> diff --git a/drivers/media/platform/qcom/venus/hfi.h >>> b/drivers/media/platform/qcom/venus/hfi.h >>> index f25d412d6553..bd5ca5a8b945 100644 >>> --- a/drivers/media/platform/qcom/venus/hfi.h >>> +++ b/drivers/media/platform/qcom/venus/hfi.h >>> @@ -10,6 +10,8 @@ >>> >>> #include "hfi_helper.h" >>> >>> +#define VB2_MAX_FRAME 32 >>> + >>> #define VIDC_SESSION_TYPE_VPE 0 >>> #define VIDC_SESSION_TYPE_ENC 1 >>> #define VIDC_SESSION_TYPE_DEC 2 >>> diff --git a/drivers/media/platform/verisilicon/hantro_hw.h >>> b/drivers/media/platform/verisilicon/hantro_hw.h >>> index e83f0c523a30..9e8faf7ba6fb 100644 >>> --- a/drivers/media/platform/verisilicon/hantro_hw.h >>> +++ b/drivers/media/platform/verisilicon/hantro_hw.h >>> @@ -15,6 +15,8 @@ >>> #include <media/v4l2-vp9.h> >>> #include <media/videobuf2-core.h> >>> >>> +#define VB2_MAX_FRAME 32 >>> + >>> #define DEC_8190_ALIGN_MASK 0x07U >>> >>> #define MB_DIM 16 >>> diff --git a/drivers/staging/media/ipu3/ipu3-v4l2.c >>> b/drivers/staging/media/ipu3/ipu3-v4l2.c >>> index e530767e80a5..6627b5c2d4d6 100644 >>> --- a/drivers/staging/media/ipu3/ipu3-v4l2.c >>> +++ b/drivers/staging/media/ipu3/ipu3-v4l2.c >>> @@ -10,6 +10,8 @@ >>> #include "ipu3.h" >>> #include "ipu3-dmamap.h" >>> >>> +#define VB2_MAX_FRAME 32 >>> + >>> /******************** v4l2_subdev_ops ********************/ >>> >>> #define IPU3_RUNNING_MODE_VIDEO 0 >>> diff --git a/include/media/videobuf2-core.h >>> b/include/media/videobuf2-core.h >>> index 77921cf894ef..080b783d608d 100644 >>> --- a/include/media/videobuf2-core.h >>> +++ b/include/media/videobuf2-core.h >>> @@ -20,7 +20,6 @@ >>> #include <media/media-request.h> >>> #include <media/frame_vector.h> >>> >>> -#define VB2_MAX_FRAME (32) >>> #define VB2_MAX_PLANES (8) >>> >>> /** >>> diff --git a/include/media/videobuf2-v4l2.h >>> b/include/media/videobuf2-v4l2.h >>> index 5a845887850b..88a7a565170e 100644 >>> --- a/include/media/videobuf2-v4l2.h >>> +++ b/include/media/videobuf2-v4l2.h >>> @@ -15,10 +15,6 @@ >>> #include <linux/videodev2.h> >>> #include <media/videobuf2-core.h> >>> >>> -#if VB2_MAX_FRAME != VIDEO_MAX_FRAME >>> -#error VB2_MAX_FRAME != VIDEO_MAX_FRAME >>> -#endif >>> - >>> #if VB2_MAX_PLANES != VIDEO_MAX_PLANES >>> #error VB2_MAX_PLANES != VIDEO_MAX_PLANES >>> #endif >>> -- >>> 2.39.2 >>> -- Hsia-Jun(Randy) Li ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v3 04/11] media: videobuf2: Stop define VB2_MAX_FRAME as global 2023-07-03 8:35 ` Hsia-Jun Li @ 2023-07-03 10:53 ` Benjamin Gaignard 2023-07-03 11:05 ` Hsia-Jun Li 2023-07-12 10:48 ` Tomasz Figa 1 sibling, 1 reply; 27+ messages in thread From: Benjamin Gaignard @ 2023-07-03 10:53 UTC (permalink / raw) To: Hsia-Jun Li Cc: linux-media, linux-kernel, linux-arm-kernel, m.szyprowski, linux-mediatek, linux-arm-msm, hverkuil-cisco, ezequiel, p.zabel, linux-rockchip, mchehab, linux-staging, ming.qian, kernel, gregkh, tfiga, nicolas.dufresne Le 03/07/2023 à 10:35, Hsia-Jun Li a écrit : > > On 7/3/23 16:09, Benjamin Gaignard wrote: >> CAUTION: Email originated externally, do not click links or open >> attachments unless you recognize the sender and know the content is >> safe. >> >> >> Le 30/06/2023 à 11:51, Hsia-Jun Li a écrit : >>> >>> On 6/22/23 21:13, Benjamin Gaignard wrote: >>>> CAUTION: Email originated externally, do not click links or open >>>> attachments unless you recognize the sender and know the content is >>>> safe. >>>> >>>> >>>> After changing bufs arrays to a dynamic allocated array >>>> VB2_MAX_FRAME doesn't mean anything for videobuf2 core. >>> >>> I think make it 64 which is the VB2_MAX_FRAME in Android GKI kernel is >>> more reasonable. >>> >>> It would be hard to iterate the whole array, it would go worse with a >>> filter. Such iterate may need to go twice because you mix >>> post-processing buffer and decoding buffer(with MV) in the same array. >> >> Here I don't want to change drivers behavior so I keep the same value. >> If it happens that they need more buffers, like for dynamic >> resolution change >> feature for Verisilicon VP9 decoder, case by case patches will be >> needed. >> > I just don't like the idea that using a variant length array here. As far I have understand Hans and/or Laurent were happy to use that but I may have misunderstood them. > > And I could explain why you won't need so many buffers for the > performance of decoding. > > VP9 could support 10 reference frames in dpb. > > Even for those frequent resolution changing test set, it would only > happen to two resolutions, > > 32 would be enough for 20 buffers of two resolution plus golden > frames. It also leaves enough slots for re-order latency. > > If your case had more two resolutions, likes low->medium->high. VP9 header doesn't tell you that video resolution will change and in which way. It cloud be from factor x1/2 to x16 and multiple time so you can use lot of (too much) buffers. > > I would suggest just skip the medium resolutions, just allocate the > lower one first for fast playback then the highest for all the possible > > medium cases. Reallocation happens frequently would only cause memory > fragment, nothing benefits your performance. > >> >>> >>>> Remove it from the core definitions but keep it for drivers internal >>>> needs. >>>> >>>> Signed-off-by: Benjamin Gaignard <benjamin.gaignard@collabora.com> >>>> --- >>>> drivers/media/common/videobuf2/videobuf2-core.c | 2 ++ >>>> drivers/media/platform/amphion/vdec.c | 1 + >>>> .../media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c | 2 ++ >>>> drivers/media/platform/qcom/venus/hfi.h | 2 ++ >>>> drivers/media/platform/verisilicon/hantro_hw.h | 2 ++ >>>> drivers/staging/media/ipu3/ipu3-v4l2.c | 2 ++ >>>> include/media/videobuf2-core.h | 1 - >>>> include/media/videobuf2-v4l2.h | 4 ---- >>>> 8 files changed, 11 insertions(+), 5 deletions(-) >>>> >>>> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c >>>> b/drivers/media/common/videobuf2/videobuf2-core.c >>>> index 86e1e926fa45..899783f67580 100644 >>>> --- a/drivers/media/common/videobuf2/videobuf2-core.c >>>> +++ b/drivers/media/common/videobuf2/videobuf2-core.c >>>> @@ -31,6 +31,8 @@ >>>> >>>> #include <trace/events/vb2.h> >>>> >>>> +#define VB2_MAX_FRAME 32 >>>> + >>>> static int debug; >>>> module_param(debug, int, 0644); >>>> >>>> diff --git a/drivers/media/platform/amphion/vdec.c >>>> b/drivers/media/platform/amphion/vdec.c >>>> index 3fa1a74a2e20..b3219f6d17fa 100644 >>>> --- a/drivers/media/platform/amphion/vdec.c >>>> +++ b/drivers/media/platform/amphion/vdec.c >>>> @@ -28,6 +28,7 @@ >>>> >>>> #define VDEC_MIN_BUFFER_CAP 8 >>>> #define VDEC_MIN_BUFFER_OUT 8 >>>> +#define VB2_MAX_FRAME 32 >>>> >>>> struct vdec_fs_info { >>>> char name[8]; >>>> diff --git >>>> a/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >>>> b/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >>>> index 6532a69f1fa8..a1e0f24bb91c 100644 >>>> --- >>>> a/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >>>> +++ >>>> b/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >>>> @@ -16,6 +16,8 @@ >>>> #include "../vdec_drv_if.h" >>>> #include "../vdec_vpu_if.h" >>>> >>>> +#define VB2_MAX_FRAME 32 >>>> + >>>> /* reset_frame_context defined in VP9 spec */ >>>> #define VP9_RESET_FRAME_CONTEXT_NONE0 0 >>>> #define VP9_RESET_FRAME_CONTEXT_NONE1 1 >>>> diff --git a/drivers/media/platform/qcom/venus/hfi.h >>>> b/drivers/media/platform/qcom/venus/hfi.h >>>> index f25d412d6553..bd5ca5a8b945 100644 >>>> --- a/drivers/media/platform/qcom/venus/hfi.h >>>> +++ b/drivers/media/platform/qcom/venus/hfi.h >>>> @@ -10,6 +10,8 @@ >>>> >>>> #include "hfi_helper.h" >>>> >>>> +#define VB2_MAX_FRAME 32 >>>> + >>>> #define VIDC_SESSION_TYPE_VPE 0 >>>> #define VIDC_SESSION_TYPE_ENC 1 >>>> #define VIDC_SESSION_TYPE_DEC 2 >>>> diff --git a/drivers/media/platform/verisilicon/hantro_hw.h >>>> b/drivers/media/platform/verisilicon/hantro_hw.h >>>> index e83f0c523a30..9e8faf7ba6fb 100644 >>>> --- a/drivers/media/platform/verisilicon/hantro_hw.h >>>> +++ b/drivers/media/platform/verisilicon/hantro_hw.h >>>> @@ -15,6 +15,8 @@ >>>> #include <media/v4l2-vp9.h> >>>> #include <media/videobuf2-core.h> >>>> >>>> +#define VB2_MAX_FRAME 32 >>>> + >>>> #define DEC_8190_ALIGN_MASK 0x07U >>>> >>>> #define MB_DIM 16 >>>> diff --git a/drivers/staging/media/ipu3/ipu3-v4l2.c >>>> b/drivers/staging/media/ipu3/ipu3-v4l2.c >>>> index e530767e80a5..6627b5c2d4d6 100644 >>>> --- a/drivers/staging/media/ipu3/ipu3-v4l2.c >>>> +++ b/drivers/staging/media/ipu3/ipu3-v4l2.c >>>> @@ -10,6 +10,8 @@ >>>> #include "ipu3.h" >>>> #include "ipu3-dmamap.h" >>>> >>>> +#define VB2_MAX_FRAME 32 >>>> + >>>> /******************** v4l2_subdev_ops ********************/ >>>> >>>> #define IPU3_RUNNING_MODE_VIDEO 0 >>>> diff --git a/include/media/videobuf2-core.h >>>> b/include/media/videobuf2-core.h >>>> index 77921cf894ef..080b783d608d 100644 >>>> --- a/include/media/videobuf2-core.h >>>> +++ b/include/media/videobuf2-core.h >>>> @@ -20,7 +20,6 @@ >>>> #include <media/media-request.h> >>>> #include <media/frame_vector.h> >>>> >>>> -#define VB2_MAX_FRAME (32) >>>> #define VB2_MAX_PLANES (8) >>>> >>>> /** >>>> diff --git a/include/media/videobuf2-v4l2.h >>>> b/include/media/videobuf2-v4l2.h >>>> index 5a845887850b..88a7a565170e 100644 >>>> --- a/include/media/videobuf2-v4l2.h >>>> +++ b/include/media/videobuf2-v4l2.h >>>> @@ -15,10 +15,6 @@ >>>> #include <linux/videodev2.h> >>>> #include <media/videobuf2-core.h> >>>> >>>> -#if VB2_MAX_FRAME != VIDEO_MAX_FRAME >>>> -#error VB2_MAX_FRAME != VIDEO_MAX_FRAME >>>> -#endif >>>> - >>>> #if VB2_MAX_PLANES != VIDEO_MAX_PLANES >>>> #error VB2_MAX_PLANES != VIDEO_MAX_PLANES >>>> #endif >>>> -- >>>> 2.39.2 >>>> ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v3 04/11] media: videobuf2: Stop define VB2_MAX_FRAME as global 2023-07-03 10:53 ` Benjamin Gaignard @ 2023-07-03 11:05 ` Hsia-Jun Li 0 siblings, 0 replies; 27+ messages in thread From: Hsia-Jun Li @ 2023-07-03 11:05 UTC (permalink / raw) To: Benjamin Gaignard Cc: linux-media, linux-kernel, linux-arm-kernel, m.szyprowski, linux-mediatek, linux-arm-msm, hverkuil-cisco, ezequiel, p.zabel, linux-rockchip, mchehab, linux-staging, ming.qian, kernel, gregkh, tfiga, nicolas.dufresne On 7/3/23 18:53, Benjamin Gaignard wrote: > CAUTION: Email originated externally, do not click links or open > attachments unless you recognize the sender and know the content is safe. > > > Le 03/07/2023 à 10:35, Hsia-Jun Li a écrit : >> >> On 7/3/23 16:09, Benjamin Gaignard wrote: >>> CAUTION: Email originated externally, do not click links or open >>> attachments unless you recognize the sender and know the content is >>> safe. >>> >>> >>> Le 30/06/2023 à 11:51, Hsia-Jun Li a écrit : >>>> >>>> On 6/22/23 21:13, Benjamin Gaignard wrote: >>>>> CAUTION: Email originated externally, do not click links or open >>>>> attachments unless you recognize the sender and know the content is >>>>> safe. >>>>> >>>>> >>>>> After changing bufs arrays to a dynamic allocated array >>>>> VB2_MAX_FRAME doesn't mean anything for videobuf2 core. >>>> >>>> I think make it 64 which is the VB2_MAX_FRAME in Android GKI kernel is >>>> more reasonable. >>>> >>>> It would be hard to iterate the whole array, it would go worse with a >>>> filter. Such iterate may need to go twice because you mix >>>> post-processing buffer and decoding buffer(with MV) in the same array. >>> >>> Here I don't want to change drivers behavior so I keep the same value. >>> If it happens that they need more buffers, like for dynamic >>> resolution change >>> feature for Verisilicon VP9 decoder, case by case patches will be >>> needed. >>> >> I just don't like the idea that using a variant length array here. > > As far I have understand Hans and/or Laurent were happy to use that > but I may have misunderstood them. > >> >> And I could explain why you won't need so many buffers for the >> performance of decoding. >> >> VP9 could support 10 reference frames in dpb. >> >> Even for those frequent resolution changing test set, it would only >> happen to two resolutions, >> >> 32 would be enough for 20 buffers of two resolution plus golden >> frames. It also leaves enough slots for re-order latency. >> >> If your case had more two resolutions, likes low->medium->high. > > VP9 header doesn't tell you that video resolution will change and in > which way. > It cloud be from factor x1/2 to x16 and multiple time so you can use > lot of (too much) buffers. I know VP9 doesn't have sequence header. I think you are talking about scaling frame, which should be allocated when it is requested. And this can't break the max reference capability requirement of VP9. > >> >> I would suggest just skip the medium resolutions, just allocate the >> lower one first for fast playback then the highest for all the possible >> >> medium cases. Reallocation happens frequently would only cause memory >> fragment, nothing benefits your performance. >> >>> >>>> >>>>> Remove it from the core definitions but keep it for drivers internal >>>>> needs. >>>>> >>>>> Signed-off-by: Benjamin Gaignard <benjamin.gaignard@collabora.com> >>>>> --- >>>>> drivers/media/common/videobuf2/videobuf2-core.c | 2 ++ >>>>> drivers/media/platform/amphion/vdec.c | 1 + >>>>> .../media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c | 2 ++ >>>>> drivers/media/platform/qcom/venus/hfi.h | 2 ++ >>>>> drivers/media/platform/verisilicon/hantro_hw.h | 2 ++ >>>>> drivers/staging/media/ipu3/ipu3-v4l2.c | 2 ++ >>>>> include/media/videobuf2-core.h | 1 - >>>>> include/media/videobuf2-v4l2.h | 4 ---- >>>>> 8 files changed, 11 insertions(+), 5 deletions(-) >>>>> >>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c >>>>> b/drivers/media/common/videobuf2/videobuf2-core.c >>>>> index 86e1e926fa45..899783f67580 100644 >>>>> --- a/drivers/media/common/videobuf2/videobuf2-core.c >>>>> +++ b/drivers/media/common/videobuf2/videobuf2-core.c >>>>> @@ -31,6 +31,8 @@ >>>>> >>>>> #include <trace/events/vb2.h> >>>>> >>>>> +#define VB2_MAX_FRAME 32 >>>>> + >>>>> static int debug; >>>>> module_param(debug, int, 0644); >>>>> >>>>> diff --git a/drivers/media/platform/amphion/vdec.c >>>>> b/drivers/media/platform/amphion/vdec.c >>>>> index 3fa1a74a2e20..b3219f6d17fa 100644 >>>>> --- a/drivers/media/platform/amphion/vdec.c >>>>> +++ b/drivers/media/platform/amphion/vdec.c >>>>> @@ -28,6 +28,7 @@ >>>>> >>>>> #define VDEC_MIN_BUFFER_CAP 8 >>>>> #define VDEC_MIN_BUFFER_OUT 8 >>>>> +#define VB2_MAX_FRAME 32 >>>>> >>>>> struct vdec_fs_info { >>>>> char name[8]; >>>>> diff --git >>>>> a/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >>>>> b/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >>>>> index 6532a69f1fa8..a1e0f24bb91c 100644 >>>>> --- >>>>> a/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >>>>> +++ >>>>> b/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >>>>> @@ -16,6 +16,8 @@ >>>>> #include "../vdec_drv_if.h" >>>>> #include "../vdec_vpu_if.h" >>>>> >>>>> +#define VB2_MAX_FRAME 32 >>>>> + >>>>> /* reset_frame_context defined in VP9 spec */ >>>>> #define VP9_RESET_FRAME_CONTEXT_NONE0 0 >>>>> #define VP9_RESET_FRAME_CONTEXT_NONE1 1 >>>>> diff --git a/drivers/media/platform/qcom/venus/hfi.h >>>>> b/drivers/media/platform/qcom/venus/hfi.h >>>>> index f25d412d6553..bd5ca5a8b945 100644 >>>>> --- a/drivers/media/platform/qcom/venus/hfi.h >>>>> +++ b/drivers/media/platform/qcom/venus/hfi.h >>>>> @@ -10,6 +10,8 @@ >>>>> >>>>> #include "hfi_helper.h" >>>>> >>>>> +#define VB2_MAX_FRAME 32 >>>>> + >>>>> #define VIDC_SESSION_TYPE_VPE 0 >>>>> #define VIDC_SESSION_TYPE_ENC 1 >>>>> #define VIDC_SESSION_TYPE_DEC 2 >>>>> diff --git a/drivers/media/platform/verisilicon/hantro_hw.h >>>>> b/drivers/media/platform/verisilicon/hantro_hw.h >>>>> index e83f0c523a30..9e8faf7ba6fb 100644 >>>>> --- a/drivers/media/platform/verisilicon/hantro_hw.h >>>>> +++ b/drivers/media/platform/verisilicon/hantro_hw.h >>>>> @@ -15,6 +15,8 @@ >>>>> #include <media/v4l2-vp9.h> >>>>> #include <media/videobuf2-core.h> >>>>> >>>>> +#define VB2_MAX_FRAME 32 >>>>> + >>>>> #define DEC_8190_ALIGN_MASK 0x07U >>>>> >>>>> #define MB_DIM 16 >>>>> diff --git a/drivers/staging/media/ipu3/ipu3-v4l2.c >>>>> b/drivers/staging/media/ipu3/ipu3-v4l2.c >>>>> index e530767e80a5..6627b5c2d4d6 100644 >>>>> --- a/drivers/staging/media/ipu3/ipu3-v4l2.c >>>>> +++ b/drivers/staging/media/ipu3/ipu3-v4l2.c >>>>> @@ -10,6 +10,8 @@ >>>>> #include "ipu3.h" >>>>> #include "ipu3-dmamap.h" >>>>> >>>>> +#define VB2_MAX_FRAME 32 >>>>> + >>>>> /******************** v4l2_subdev_ops ********************/ >>>>> >>>>> #define IPU3_RUNNING_MODE_VIDEO 0 >>>>> diff --git a/include/media/videobuf2-core.h >>>>> b/include/media/videobuf2-core.h >>>>> index 77921cf894ef..080b783d608d 100644 >>>>> --- a/include/media/videobuf2-core.h >>>>> +++ b/include/media/videobuf2-core.h >>>>> @@ -20,7 +20,6 @@ >>>>> #include <media/media-request.h> >>>>> #include <media/frame_vector.h> >>>>> >>>>> -#define VB2_MAX_FRAME (32) >>>>> #define VB2_MAX_PLANES (8) >>>>> >>>>> /** >>>>> diff --git a/include/media/videobuf2-v4l2.h >>>>> b/include/media/videobuf2-v4l2.h >>>>> index 5a845887850b..88a7a565170e 100644 >>>>> --- a/include/media/videobuf2-v4l2.h >>>>> +++ b/include/media/videobuf2-v4l2.h >>>>> @@ -15,10 +15,6 @@ >>>>> #include <linux/videodev2.h> >>>>> #include <media/videobuf2-core.h> >>>>> >>>>> -#if VB2_MAX_FRAME != VIDEO_MAX_FRAME >>>>> -#error VB2_MAX_FRAME != VIDEO_MAX_FRAME >>>>> -#endif >>>>> - >>>>> #if VB2_MAX_PLANES != VIDEO_MAX_PLANES >>>>> #error VB2_MAX_PLANES != VIDEO_MAX_PLANES >>>>> #endif >>>>> -- >>>>> 2.39.2 >>>>> -- Hsia-Jun(Randy) Li ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v3 04/11] media: videobuf2: Stop define VB2_MAX_FRAME as global 2023-07-03 8:35 ` Hsia-Jun Li 2023-07-03 10:53 ` Benjamin Gaignard @ 2023-07-12 10:48 ` Tomasz Figa 2023-07-17 7:44 ` Hsia-Jun Li 1 sibling, 1 reply; 27+ messages in thread From: Tomasz Figa @ 2023-07-12 10:48 UTC (permalink / raw) To: Hsia-Jun Li Cc: Benjamin Gaignard, linux-media, linux-kernel, linux-arm-kernel, m.szyprowski, linux-mediatek, linux-arm-msm, hverkuil-cisco, ezequiel, p.zabel, linux-rockchip, mchehab, linux-staging, ming.qian, kernel, gregkh, nicolas.dufresne On Mon, Jul 03, 2023 at 04:35:30PM +0800, Hsia-Jun Li wrote: > > On 7/3/23 16:09, Benjamin Gaignard wrote: > > CAUTION: Email originated externally, do not click links or open > > attachments unless you recognize the sender and know the content is > > safe. > > > > > > Le 30/06/2023 à 11:51, Hsia-Jun Li a écrit : > > > > > > On 6/22/23 21:13, Benjamin Gaignard wrote: > > > > CAUTION: Email originated externally, do not click links or open > > > > attachments unless you recognize the sender and know the content is > > > > safe. > > > > > > > > > > > > After changing bufs arrays to a dynamic allocated array > > > > VB2_MAX_FRAME doesn't mean anything for videobuf2 core. > > > > > > I think make it 64 which is the VB2_MAX_FRAME in Android GKI kernel is > > > more reasonable. > > > > > > It would be hard to iterate the whole array, it would go worse with a > > > filter. Such iterate may need to go twice because you mix > > > post-processing buffer and decoding buffer(with MV) in the same array. > > > > Here I don't want to change drivers behavior so I keep the same value. > > If it happens that they need more buffers, like for dynamic resolution > > change > > feature for Verisilicon VP9 decoder, case by case patches will be needed. > > > I just don't like the idea that using a variant length array here. > "I don't like" is not an argument. We had a number of arguments for using a generic helper (originally idr, but later decided to go with XArray, because the former is now deprecated) that we pointed out in our review comments for previous revisions. It wasn't really about the size being variable, but rather avoiding open coding things in vb2 and duplicating what's already implemented in generic code. > And I could explain why you won't need so many buffers for the performance > of decoding. > > VP9 could support 10 reference frames in dpb. > > Even for those frequent resolution changing test set, it would only happen > to two resolutions, > > 32 would be enough for 20 buffers of two resolution plus golden frames. It > also leaves enough slots for re-order latency. > > If your case had more two resolutions, likes low->medium->high. > > I would suggest just skip the medium resolutions, just allocate the lower > one first for fast playback then the highest for all the possible > > medium cases. Reallocation happens frequently would only cause memory > fragment, nothing benefits your performance. > We have mechanisms in the kernel to deal with memory fragmentation (migration/compaction) and it would still only matters for the pathologic cases of hardware that require physically contiguous memory. Modern hardware with proper DMA capabilities can either scatter-gather or are equipped with an IOMMU, so the allocation always happens in page granularity and fragmentation is avoided. Best regards, Tomasz > > > > > > > > > Remove it from the core definitions but keep it for drivers internal > > > > needs. > > > > > > > > Signed-off-by: Benjamin Gaignard <benjamin.gaignard@collabora.com> > > > > --- > > > > drivers/media/common/videobuf2/videobuf2-core.c | 2 ++ > > > > drivers/media/platform/amphion/vdec.c | 1 + > > > > .../media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c | 2 ++ > > > > drivers/media/platform/qcom/venus/hfi.h | 2 ++ > > > > drivers/media/platform/verisilicon/hantro_hw.h | 2 ++ > > > > drivers/staging/media/ipu3/ipu3-v4l2.c | 2 ++ > > > > include/media/videobuf2-core.h | 1 - > > > > include/media/videobuf2-v4l2.h | 4 ---- > > > > 8 files changed, 11 insertions(+), 5 deletions(-) > > > > > > > > diff --git a/drivers/media/common/videobuf2/videobuf2-core.c > > > > b/drivers/media/common/videobuf2/videobuf2-core.c > > > > index 86e1e926fa45..899783f67580 100644 > > > > --- a/drivers/media/common/videobuf2/videobuf2-core.c > > > > +++ b/drivers/media/common/videobuf2/videobuf2-core.c > > > > @@ -31,6 +31,8 @@ > > > > > > > > #include <trace/events/vb2.h> > > > > > > > > +#define VB2_MAX_FRAME 32 > > > > + > > > > static int debug; > > > > module_param(debug, int, 0644); > > > > > > > > diff --git a/drivers/media/platform/amphion/vdec.c > > > > b/drivers/media/platform/amphion/vdec.c > > > > index 3fa1a74a2e20..b3219f6d17fa 100644 > > > > --- a/drivers/media/platform/amphion/vdec.c > > > > +++ b/drivers/media/platform/amphion/vdec.c > > > > @@ -28,6 +28,7 @@ > > > > > > > > #define VDEC_MIN_BUFFER_CAP 8 > > > > #define VDEC_MIN_BUFFER_OUT 8 > > > > +#define VB2_MAX_FRAME 32 > > > > > > > > struct vdec_fs_info { > > > > char name[8]; > > > > diff --git > > > > a/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c > > > > b/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c > > > > index 6532a69f1fa8..a1e0f24bb91c 100644 > > > > --- a/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c > > > > +++ b/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c > > > > @@ -16,6 +16,8 @@ > > > > #include "../vdec_drv_if.h" > > > > #include "../vdec_vpu_if.h" > > > > > > > > +#define VB2_MAX_FRAME 32 > > > > + > > > > /* reset_frame_context defined in VP9 spec */ > > > > #define VP9_RESET_FRAME_CONTEXT_NONE0 0 > > > > #define VP9_RESET_FRAME_CONTEXT_NONE1 1 > > > > diff --git a/drivers/media/platform/qcom/venus/hfi.h > > > > b/drivers/media/platform/qcom/venus/hfi.h > > > > index f25d412d6553..bd5ca5a8b945 100644 > > > > --- a/drivers/media/platform/qcom/venus/hfi.h > > > > +++ b/drivers/media/platform/qcom/venus/hfi.h > > > > @@ -10,6 +10,8 @@ > > > > > > > > #include "hfi_helper.h" > > > > > > > > +#define VB2_MAX_FRAME 32 > > > > + > > > > #define VIDC_SESSION_TYPE_VPE 0 > > > > #define VIDC_SESSION_TYPE_ENC 1 > > > > #define VIDC_SESSION_TYPE_DEC 2 > > > > diff --git a/drivers/media/platform/verisilicon/hantro_hw.h > > > > b/drivers/media/platform/verisilicon/hantro_hw.h > > > > index e83f0c523a30..9e8faf7ba6fb 100644 > > > > --- a/drivers/media/platform/verisilicon/hantro_hw.h > > > > +++ b/drivers/media/platform/verisilicon/hantro_hw.h > > > > @@ -15,6 +15,8 @@ > > > > #include <media/v4l2-vp9.h> > > > > #include <media/videobuf2-core.h> > > > > > > > > +#define VB2_MAX_FRAME 32 > > > > + > > > > #define DEC_8190_ALIGN_MASK 0x07U > > > > > > > > #define MB_DIM 16 > > > > diff --git a/drivers/staging/media/ipu3/ipu3-v4l2.c > > > > b/drivers/staging/media/ipu3/ipu3-v4l2.c > > > > index e530767e80a5..6627b5c2d4d6 100644 > > > > --- a/drivers/staging/media/ipu3/ipu3-v4l2.c > > > > +++ b/drivers/staging/media/ipu3/ipu3-v4l2.c > > > > @@ -10,6 +10,8 @@ > > > > #include "ipu3.h" > > > > #include "ipu3-dmamap.h" > > > > > > > > +#define VB2_MAX_FRAME 32 > > > > + > > > > /******************** v4l2_subdev_ops ********************/ > > > > > > > > #define IPU3_RUNNING_MODE_VIDEO 0 > > > > diff --git a/include/media/videobuf2-core.h > > > > b/include/media/videobuf2-core.h > > > > index 77921cf894ef..080b783d608d 100644 > > > > --- a/include/media/videobuf2-core.h > > > > +++ b/include/media/videobuf2-core.h > > > > @@ -20,7 +20,6 @@ > > > > #include <media/media-request.h> > > > > #include <media/frame_vector.h> > > > > > > > > -#define VB2_MAX_FRAME (32) > > > > #define VB2_MAX_PLANES (8) > > > > > > > > /** > > > > diff --git a/include/media/videobuf2-v4l2.h > > > > b/include/media/videobuf2-v4l2.h > > > > index 5a845887850b..88a7a565170e 100644 > > > > --- a/include/media/videobuf2-v4l2.h > > > > +++ b/include/media/videobuf2-v4l2.h > > > > @@ -15,10 +15,6 @@ > > > > #include <linux/videodev2.h> > > > > #include <media/videobuf2-core.h> > > > > > > > > -#if VB2_MAX_FRAME != VIDEO_MAX_FRAME > > > > -#error VB2_MAX_FRAME != VIDEO_MAX_FRAME > > > > -#endif > > > > - > > > > #if VB2_MAX_PLANES != VIDEO_MAX_PLANES > > > > #error VB2_MAX_PLANES != VIDEO_MAX_PLANES > > > > #endif > > > > -- > > > > 2.39.2 > > > > > -- > Hsia-Jun(Randy) Li > ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v3 04/11] media: videobuf2: Stop define VB2_MAX_FRAME as global 2023-07-12 10:48 ` Tomasz Figa @ 2023-07-17 7:44 ` Hsia-Jun Li 2023-07-28 6:46 ` Tomasz Figa 0 siblings, 1 reply; 27+ messages in thread From: Hsia-Jun Li @ 2023-07-17 7:44 UTC (permalink / raw) To: Tomasz Figa Cc: Benjamin Gaignard, linux-media, linux-kernel, linux-arm-kernel, m.szyprowski, linux-mediatek, linux-arm-msm, hverkuil-cisco, ezequiel, p.zabel, linux-rockchip, mchehab, linux-staging, ming.qian, kernel, gregkh, nicolas.dufresne On 7/12/23 18:48, Tomasz Figa wrote: > CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe. > > > On Mon, Jul 03, 2023 at 04:35:30PM +0800, Hsia-Jun Li wrote: >> On 7/3/23 16:09, Benjamin Gaignard wrote: >>> CAUTION: Email originated externally, do not click links or open >>> attachments unless you recognize the sender and know the content is >>> safe. >>> >>> >>> Le 30/06/2023 à 11:51, Hsia-Jun Li a écrit : >>>> On 6/22/23 21:13, Benjamin Gaignard wrote: >>>>> CAUTION: Email originated externally, do not click links or open >>>>> attachments unless you recognize the sender and know the content is >>>>> safe. >>>>> >>>>> >>>>> After changing bufs arrays to a dynamic allocated array >>>>> VB2_MAX_FRAME doesn't mean anything for videobuf2 core. >>>> I think make it 64 which is the VB2_MAX_FRAME in Android GKI kernel is >>>> more reasonable. >>>> >>>> It would be hard to iterate the whole array, it would go worse with a >>>> filter. Such iterate may need to go twice because you mix >>>> post-processing buffer and decoding buffer(with MV) in the same array. >>> Here I don't want to change drivers behavior so I keep the same value. >>> If it happens that they need more buffers, like for dynamic resolution >>> change >>> feature for Verisilicon VP9 decoder, case by case patches will be needed. >>> >> I just don't like the idea that using a variant length array here. >> > "I don't like" is not an argument. We had a number of arguments for > using a generic helper (originally idr, but later decided to go with > XArray, because the former is now deprecated) that we pointed out in > our review comments for previous revisions. It wasn't really about the > size being variable, but rather avoiding open coding things in vb2 and > duplicating what's already implemented in generic code. I just want to say I don't think we need a variable length array to store the buffer here. And the below is the reason that such a case could be avoided in the first place. > >> And I could explain why you won't need so many buffers for the performance >> of decoding. >> >> VP9 could support 10 reference frames in dpb. >> >> Even for those frequent resolution changing test set, it would only happen >> to two resolutions, >> >> 32 would be enough for 20 buffers of two resolution plus golden frames. It >> also leaves enough slots for re-order latency. >> >> If your case had more two resolutions, likes low->medium->high. >> >> I would suggest just skip the medium resolutions, just allocate the lower >> one first for fast playback then the highest for all the possible >> >> medium cases. Reallocation happens frequently would only cause memory >> fragment, nothing benefits your performance. >> > We have mechanisms in the kernel to deal with memory fragmentation > (migration/compaction) and it would still only matters for the > pathologic cases of hardware that require physically contiguous memory. > Modern hardware with proper DMA capabilities can either scatter-gather > or are equipped with an IOMMU, so the allocation always happens in page > granularity and fragmentation is avoided. Unfortunately, there are more devices that didn't have a IOMMU attached to it, supporting scatter gather is more odd. It would be more likely that IOMMU would be disabled for the performance reason. > > Best regards, > Tomasz > >>>>> Remove it from the core definitions but keep it for drivers internal >>>>> needs. >>>>> >>>>> Signed-off-by: Benjamin Gaignard <benjamin.gaignard@collabora.com> >>>>> --- >>>>> drivers/media/common/videobuf2/videobuf2-core.c | 2 ++ >>>>> drivers/media/platform/amphion/vdec.c | 1 + >>>>> .../media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c | 2 ++ >>>>> drivers/media/platform/qcom/venus/hfi.h | 2 ++ >>>>> drivers/media/platform/verisilicon/hantro_hw.h | 2 ++ >>>>> drivers/staging/media/ipu3/ipu3-v4l2.c | 2 ++ >>>>> include/media/videobuf2-core.h | 1 - >>>>> include/media/videobuf2-v4l2.h | 4 ---- >>>>> 8 files changed, 11 insertions(+), 5 deletions(-) >>>>> >>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c >>>>> b/drivers/media/common/videobuf2/videobuf2-core.c >>>>> index 86e1e926fa45..899783f67580 100644 >>>>> --- a/drivers/media/common/videobuf2/videobuf2-core.c >>>>> +++ b/drivers/media/common/videobuf2/videobuf2-core.c >>>>> @@ -31,6 +31,8 @@ >>>>> >>>>> #include <trace/events/vb2.h> >>>>> >>>>> +#define VB2_MAX_FRAME 32 >>>>> + >>>>> static int debug; >>>>> module_param(debug, int, 0644); >>>>> >>>>> diff --git a/drivers/media/platform/amphion/vdec.c >>>>> b/drivers/media/platform/amphion/vdec.c >>>>> index 3fa1a74a2e20..b3219f6d17fa 100644 >>>>> --- a/drivers/media/platform/amphion/vdec.c >>>>> +++ b/drivers/media/platform/amphion/vdec.c >>>>> @@ -28,6 +28,7 @@ >>>>> >>>>> #define VDEC_MIN_BUFFER_CAP 8 >>>>> #define VDEC_MIN_BUFFER_OUT 8 >>>>> +#define VB2_MAX_FRAME 32 >>>>> >>>>> struct vdec_fs_info { >>>>> char name[8]; >>>>> diff --git >>>>> a/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >>>>> b/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >>>>> index 6532a69f1fa8..a1e0f24bb91c 100644 >>>>> --- a/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >>>>> +++ b/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >>>>> @@ -16,6 +16,8 @@ >>>>> #include "../vdec_drv_if.h" >>>>> #include "../vdec_vpu_if.h" >>>>> >>>>> +#define VB2_MAX_FRAME 32 >>>>> + >>>>> /* reset_frame_context defined in VP9 spec */ >>>>> #define VP9_RESET_FRAME_CONTEXT_NONE0 0 >>>>> #define VP9_RESET_FRAME_CONTEXT_NONE1 1 >>>>> diff --git a/drivers/media/platform/qcom/venus/hfi.h >>>>> b/drivers/media/platform/qcom/venus/hfi.h >>>>> index f25d412d6553..bd5ca5a8b945 100644 >>>>> --- a/drivers/media/platform/qcom/venus/hfi.h >>>>> +++ b/drivers/media/platform/qcom/venus/hfi.h >>>>> @@ -10,6 +10,8 @@ >>>>> >>>>> #include "hfi_helper.h" >>>>> >>>>> +#define VB2_MAX_FRAME 32 >>>>> + >>>>> #define VIDC_SESSION_TYPE_VPE 0 >>>>> #define VIDC_SESSION_TYPE_ENC 1 >>>>> #define VIDC_SESSION_TYPE_DEC 2 >>>>> diff --git a/drivers/media/platform/verisilicon/hantro_hw.h >>>>> b/drivers/media/platform/verisilicon/hantro_hw.h >>>>> index e83f0c523a30..9e8faf7ba6fb 100644 >>>>> --- a/drivers/media/platform/verisilicon/hantro_hw.h >>>>> +++ b/drivers/media/platform/verisilicon/hantro_hw.h >>>>> @@ -15,6 +15,8 @@ >>>>> #include <media/v4l2-vp9.h> >>>>> #include <media/videobuf2-core.h> >>>>> >>>>> +#define VB2_MAX_FRAME 32 >>>>> + >>>>> #define DEC_8190_ALIGN_MASK 0x07U >>>>> >>>>> #define MB_DIM 16 >>>>> diff --git a/drivers/staging/media/ipu3/ipu3-v4l2.c >>>>> b/drivers/staging/media/ipu3/ipu3-v4l2.c >>>>> index e530767e80a5..6627b5c2d4d6 100644 >>>>> --- a/drivers/staging/media/ipu3/ipu3-v4l2.c >>>>> +++ b/drivers/staging/media/ipu3/ipu3-v4l2.c >>>>> @@ -10,6 +10,8 @@ >>>>> #include "ipu3.h" >>>>> #include "ipu3-dmamap.h" >>>>> >>>>> +#define VB2_MAX_FRAME 32 >>>>> + >>>>> /******************** v4l2_subdev_ops ********************/ >>>>> >>>>> #define IPU3_RUNNING_MODE_VIDEO 0 >>>>> diff --git a/include/media/videobuf2-core.h >>>>> b/include/media/videobuf2-core.h >>>>> index 77921cf894ef..080b783d608d 100644 >>>>> --- a/include/media/videobuf2-core.h >>>>> +++ b/include/media/videobuf2-core.h >>>>> @@ -20,7 +20,6 @@ >>>>> #include <media/media-request.h> >>>>> #include <media/frame_vector.h> >>>>> >>>>> -#define VB2_MAX_FRAME (32) >>>>> #define VB2_MAX_PLANES (8) >>>>> >>>>> /** >>>>> diff --git a/include/media/videobuf2-v4l2.h >>>>> b/include/media/videobuf2-v4l2.h >>>>> index 5a845887850b..88a7a565170e 100644 >>>>> --- a/include/media/videobuf2-v4l2.h >>>>> +++ b/include/media/videobuf2-v4l2.h >>>>> @@ -15,10 +15,6 @@ >>>>> #include <linux/videodev2.h> >>>>> #include <media/videobuf2-core.h> >>>>> >>>>> -#if VB2_MAX_FRAME != VIDEO_MAX_FRAME >>>>> -#error VB2_MAX_FRAME != VIDEO_MAX_FRAME >>>>> -#endif >>>>> - >>>>> #if VB2_MAX_PLANES != VIDEO_MAX_PLANES >>>>> #error VB2_MAX_PLANES != VIDEO_MAX_PLANES >>>>> #endif >>>>> -- >>>>> 2.39.2 >>>>> >> -- >> Hsia-Jun(Randy) Li >> -- Hsia-Jun(Randy) Li ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v3 04/11] media: videobuf2: Stop define VB2_MAX_FRAME as global 2023-07-17 7:44 ` Hsia-Jun Li @ 2023-07-28 6:46 ` Tomasz Figa 2023-07-28 6:55 ` Hsia-Jun Li 0 siblings, 1 reply; 27+ messages in thread From: Tomasz Figa @ 2023-07-28 6:46 UTC (permalink / raw) To: Hsia-Jun Li Cc: Benjamin Gaignard, linux-media, linux-kernel, linux-arm-kernel, m.szyprowski, linux-mediatek, linux-arm-msm, hverkuil-cisco, ezequiel, p.zabel, linux-rockchip, mchehab, linux-staging, ming.qian, kernel, gregkh, nicolas.dufresne On Mon, Jul 17, 2023 at 4:44 PM Hsia-Jun Li <Randy.Li@synaptics.com> wrote: > > > On 7/12/23 18:48, Tomasz Figa wrote: > > CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe. > > > > > > On Mon, Jul 03, 2023 at 04:35:30PM +0800, Hsia-Jun Li wrote: > >> On 7/3/23 16:09, Benjamin Gaignard wrote: > >>> CAUTION: Email originated externally, do not click links or open > >>> attachments unless you recognize the sender and know the content is > >>> safe. > >>> > >>> > >>> Le 30/06/2023 à 11:51, Hsia-Jun Li a écrit : > >>>> On 6/22/23 21:13, Benjamin Gaignard wrote: > >>>>> CAUTION: Email originated externally, do not click links or open > >>>>> attachments unless you recognize the sender and know the content is > >>>>> safe. > >>>>> > >>>>> > >>>>> After changing bufs arrays to a dynamic allocated array > >>>>> VB2_MAX_FRAME doesn't mean anything for videobuf2 core. > >>>> I think make it 64 which is the VB2_MAX_FRAME in Android GKI kernel is > >>>> more reasonable. > >>>> > >>>> It would be hard to iterate the whole array, it would go worse with a > >>>> filter. Such iterate may need to go twice because you mix > >>>> post-processing buffer and decoding buffer(with MV) in the same array. > >>> Here I don't want to change drivers behavior so I keep the same value. > >>> If it happens that they need more buffers, like for dynamic resolution > >>> change > >>> feature for Verisilicon VP9 decoder, case by case patches will be needed. > >>> > >> I just don't like the idea that using a variant length array here. > >> > > "I don't like" is not an argument. We had a number of arguments for > > using a generic helper (originally idr, but later decided to go with > > XArray, because the former is now deprecated) that we pointed out in > > our review comments for previous revisions. It wasn't really about the > > size being variable, but rather avoiding open coding things in vb2 and > > duplicating what's already implemented in generic code. > > I just want to say I don't think we need a variable length array to > store the buffer here. > > And the below is the reason that such a case could be avoided in the > first place. > > > > >> And I could explain why you won't need so many buffers for the performance > >> of decoding. > >> > >> VP9 could support 10 reference frames in dpb. > >> > >> Even for those frequent resolution changing test set, it would only happen > >> to two resolutions, > >> > >> 32 would be enough for 20 buffers of two resolution plus golden frames. It > >> also leaves enough slots for re-order latency. > >> > >> If your case had more two resolutions, likes low->medium->high. > >> > >> I would suggest just skip the medium resolutions, just allocate the lower > >> one first for fast playback then the highest for all the possible > >> > >> medium cases. Reallocation happens frequently would only cause memory > >> fragment, nothing benefits your performance. > >> > > We have mechanisms in the kernel to deal with memory fragmentation > > (migration/compaction) and it would still only matters for the > > pathologic cases of hardware that require physically contiguous memory. > > Modern hardware with proper DMA capabilities can either scatter-gather > > or are equipped with an IOMMU, so the allocation always happens in page > > granularity and fragmentation is avoided. > > Unfortunately, there are more devices that didn't have a IOMMU attached > to it, supporting scatter gather is more odd. > > It would be more likely that IOMMU would be disabled for the performance > reason. These days IOMMU is totally mandatory if you want to think about having any level of security in your system. Sure, there could be some systems that are completely isolated from any external environment, like some offline industry automation machines, but then arguably their running conditions would also be quite static and require very little memory re-allocation. I also don't buy the performance reason. CPUs have been behind MMUs for ages and nobody is running them with paging disabled for performance reasons. Similarly, most of the modern consumer systems (mobile phones, PCs) run with IOMMUs enabled for pretty much anything because of the security reason and they don't seem to be having any performance issues. In fact, it can improve the performance, because memory allocation is much easier and without contiguous careouts (as we used to have long ago on Android devices) the extra memory can be used for buffers and caches to improve system performance. Best regards, Tomasz > > > > > Best regards, > > Tomasz > > > >>>>> Remove it from the core definitions but keep it for drivers internal > >>>>> needs. > >>>>> > >>>>> Signed-off-by: Benjamin Gaignard <benjamin.gaignard@collabora.com> > >>>>> --- > >>>>> drivers/media/common/videobuf2/videobuf2-core.c | 2 ++ > >>>>> drivers/media/platform/amphion/vdec.c | 1 + > >>>>> .../media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c | 2 ++ > >>>>> drivers/media/platform/qcom/venus/hfi.h | 2 ++ > >>>>> drivers/media/platform/verisilicon/hantro_hw.h | 2 ++ > >>>>> drivers/staging/media/ipu3/ipu3-v4l2.c | 2 ++ > >>>>> include/media/videobuf2-core.h | 1 - > >>>>> include/media/videobuf2-v4l2.h | 4 ---- > >>>>> 8 files changed, 11 insertions(+), 5 deletions(-) > >>>>> > >>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c > >>>>> b/drivers/media/common/videobuf2/videobuf2-core.c > >>>>> index 86e1e926fa45..899783f67580 100644 > >>>>> --- a/drivers/media/common/videobuf2/videobuf2-core.c > >>>>> +++ b/drivers/media/common/videobuf2/videobuf2-core.c > >>>>> @@ -31,6 +31,8 @@ > >>>>> > >>>>> #include <trace/events/vb2.h> > >>>>> > >>>>> +#define VB2_MAX_FRAME 32 > >>>>> + > >>>>> static int debug; > >>>>> module_param(debug, int, 0644); > >>>>> > >>>>> diff --git a/drivers/media/platform/amphion/vdec.c > >>>>> b/drivers/media/platform/amphion/vdec.c > >>>>> index 3fa1a74a2e20..b3219f6d17fa 100644 > >>>>> --- a/drivers/media/platform/amphion/vdec.c > >>>>> +++ b/drivers/media/platform/amphion/vdec.c > >>>>> @@ -28,6 +28,7 @@ > >>>>> > >>>>> #define VDEC_MIN_BUFFER_CAP 8 > >>>>> #define VDEC_MIN_BUFFER_OUT 8 > >>>>> +#define VB2_MAX_FRAME 32 > >>>>> > >>>>> struct vdec_fs_info { > >>>>> char name[8]; > >>>>> diff --git > >>>>> a/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c > >>>>> b/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c > >>>>> index 6532a69f1fa8..a1e0f24bb91c 100644 > >>>>> --- a/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c > >>>>> +++ b/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c > >>>>> @@ -16,6 +16,8 @@ > >>>>> #include "../vdec_drv_if.h" > >>>>> #include "../vdec_vpu_if.h" > >>>>> > >>>>> +#define VB2_MAX_FRAME 32 > >>>>> + > >>>>> /* reset_frame_context defined in VP9 spec */ > >>>>> #define VP9_RESET_FRAME_CONTEXT_NONE0 0 > >>>>> #define VP9_RESET_FRAME_CONTEXT_NONE1 1 > >>>>> diff --git a/drivers/media/platform/qcom/venus/hfi.h > >>>>> b/drivers/media/platform/qcom/venus/hfi.h > >>>>> index f25d412d6553..bd5ca5a8b945 100644 > >>>>> --- a/drivers/media/platform/qcom/venus/hfi.h > >>>>> +++ b/drivers/media/platform/qcom/venus/hfi.h > >>>>> @@ -10,6 +10,8 @@ > >>>>> > >>>>> #include "hfi_helper.h" > >>>>> > >>>>> +#define VB2_MAX_FRAME 32 > >>>>> + > >>>>> #define VIDC_SESSION_TYPE_VPE 0 > >>>>> #define VIDC_SESSION_TYPE_ENC 1 > >>>>> #define VIDC_SESSION_TYPE_DEC 2 > >>>>> diff --git a/drivers/media/platform/verisilicon/hantro_hw.h > >>>>> b/drivers/media/platform/verisilicon/hantro_hw.h > >>>>> index e83f0c523a30..9e8faf7ba6fb 100644 > >>>>> --- a/drivers/media/platform/verisilicon/hantro_hw.h > >>>>> +++ b/drivers/media/platform/verisilicon/hantro_hw.h > >>>>> @@ -15,6 +15,8 @@ > >>>>> #include <media/v4l2-vp9.h> > >>>>> #include <media/videobuf2-core.h> > >>>>> > >>>>> +#define VB2_MAX_FRAME 32 > >>>>> + > >>>>> #define DEC_8190_ALIGN_MASK 0x07U > >>>>> > >>>>> #define MB_DIM 16 > >>>>> diff --git a/drivers/staging/media/ipu3/ipu3-v4l2.c > >>>>> b/drivers/staging/media/ipu3/ipu3-v4l2.c > >>>>> index e530767e80a5..6627b5c2d4d6 100644 > >>>>> --- a/drivers/staging/media/ipu3/ipu3-v4l2.c > >>>>> +++ b/drivers/staging/media/ipu3/ipu3-v4l2.c > >>>>> @@ -10,6 +10,8 @@ > >>>>> #include "ipu3.h" > >>>>> #include "ipu3-dmamap.h" > >>>>> > >>>>> +#define VB2_MAX_FRAME 32 > >>>>> + > >>>>> /******************** v4l2_subdev_ops ********************/ > >>>>> > >>>>> #define IPU3_RUNNING_MODE_VIDEO 0 > >>>>> diff --git a/include/media/videobuf2-core.h > >>>>> b/include/media/videobuf2-core.h > >>>>> index 77921cf894ef..080b783d608d 100644 > >>>>> --- a/include/media/videobuf2-core.h > >>>>> +++ b/include/media/videobuf2-core.h > >>>>> @@ -20,7 +20,6 @@ > >>>>> #include <media/media-request.h> > >>>>> #include <media/frame_vector.h> > >>>>> > >>>>> -#define VB2_MAX_FRAME (32) > >>>>> #define VB2_MAX_PLANES (8) > >>>>> > >>>>> /** > >>>>> diff --git a/include/media/videobuf2-v4l2.h > >>>>> b/include/media/videobuf2-v4l2.h > >>>>> index 5a845887850b..88a7a565170e 100644 > >>>>> --- a/include/media/videobuf2-v4l2.h > >>>>> +++ b/include/media/videobuf2-v4l2.h > >>>>> @@ -15,10 +15,6 @@ > >>>>> #include <linux/videodev2.h> > >>>>> #include <media/videobuf2-core.h> > >>>>> > >>>>> -#if VB2_MAX_FRAME != VIDEO_MAX_FRAME > >>>>> -#error VB2_MAX_FRAME != VIDEO_MAX_FRAME > >>>>> -#endif > >>>>> - > >>>>> #if VB2_MAX_PLANES != VIDEO_MAX_PLANES > >>>>> #error VB2_MAX_PLANES != VIDEO_MAX_PLANES > >>>>> #endif > >>>>> -- > >>>>> 2.39.2 > >>>>> > >> -- > >> Hsia-Jun(Randy) Li > >> > -- > Hsia-Jun(Randy) Li > ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v3 04/11] media: videobuf2: Stop define VB2_MAX_FRAME as global 2023-07-28 6:46 ` Tomasz Figa @ 2023-07-28 6:55 ` Hsia-Jun Li 2023-07-28 7:03 ` Tomasz Figa 0 siblings, 1 reply; 27+ messages in thread From: Hsia-Jun Li @ 2023-07-28 6:55 UTC (permalink / raw) To: Tomasz Figa Cc: Benjamin Gaignard, linux-media, linux-kernel, linux-arm-kernel, m.szyprowski, linux-mediatek, linux-arm-msm, hverkuil-cisco, ezequiel, p.zabel, linux-rockchip, mchehab, linux-staging, ming.qian, kernel, gregkh, nicolas.dufresne On 7/28/23 14:46, Tomasz Figa wrote: > CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe. > > > On Mon, Jul 17, 2023 at 4:44 PM Hsia-Jun Li <Randy.Li@synaptics.com> wrote: >> >> >> On 7/12/23 18:48, Tomasz Figa wrote: >>> CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe. >>> >>> >>> On Mon, Jul 03, 2023 at 04:35:30PM +0800, Hsia-Jun Li wrote: >>>> On 7/3/23 16:09, Benjamin Gaignard wrote: >>>>> CAUTION: Email originated externally, do not click links or open >>>>> attachments unless you recognize the sender and know the content is >>>>> safe. >>>>> >>>>> >>>>> Le 30/06/2023 à 11:51, Hsia-Jun Li a écrit : >>>>>> On 6/22/23 21:13, Benjamin Gaignard wrote: >>>>>>> CAUTION: Email originated externally, do not click links or open >>>>>>> attachments unless you recognize the sender and know the content is >>>>>>> safe. >>>>>>> >>>>>>> >>>>>>> After changing bufs arrays to a dynamic allocated array >>>>>>> VB2_MAX_FRAME doesn't mean anything for videobuf2 core. >>>>>> I think make it 64 which is the VB2_MAX_FRAME in Android GKI kernel is >>>>>> more reasonable. >>>>>> >>>>>> It would be hard to iterate the whole array, it would go worse with a >>>>>> filter. Such iterate may need to go twice because you mix >>>>>> post-processing buffer and decoding buffer(with MV) in the same array. >>>>> Here I don't want to change drivers behavior so I keep the same value. >>>>> If it happens that they need more buffers, like for dynamic resolution >>>>> change >>>>> feature for Verisilicon VP9 decoder, case by case patches will be needed. >>>>> >>>> I just don't like the idea that using a variant length array here. >>>> >>> "I don't like" is not an argument. We had a number of arguments for >>> using a generic helper (originally idr, but later decided to go with >>> XArray, because the former is now deprecated) that we pointed out in >>> our review comments for previous revisions. It wasn't really about the >>> size being variable, but rather avoiding open coding things in vb2 and >>> duplicating what's already implemented in generic code. >> >> I just want to say I don't think we need a variable length array to >> store the buffer here. >> >> And the below is the reason that such a case could be avoided in the >> first place. >> >>> >>>> And I could explain why you won't need so many buffers for the performance >>>> of decoding. >>>> >>>> VP9 could support 10 reference frames in dpb. >>>> >>>> Even for those frequent resolution changing test set, it would only happen >>>> to two resolutions, >>>> >>>> 32 would be enough for 20 buffers of two resolution plus golden frames. It >>>> also leaves enough slots for re-order latency. >>>> >>>> If your case had more two resolutions, likes low->medium->high. >>>> >>>> I would suggest just skip the medium resolutions, just allocate the lower >>>> one first for fast playback then the highest for all the possible >>>> >>>> medium cases. Reallocation happens frequently would only cause memory >>>> fragment, nothing benefits your performance. >>>> >>> We have mechanisms in the kernel to deal with memory fragmentation >>> (migration/compaction) and it would still only matters for the >>> pathologic cases of hardware that require physically contiguous memory. >>> Modern hardware with proper DMA capabilities can either scatter-gather >>> or are equipped with an IOMMU, so the allocation always happens in page >>> granularity and fragmentation is avoided. >> >> Unfortunately, there are more devices that didn't have a IOMMU attached >> to it, supporting scatter gather is more odd. >> >> It would be more likely that IOMMU would be disabled for the performance >> reason. > > These days IOMMU is totally mandatory if you want to think about > having any level of security in your system. Sure, there could be some > systems that are completely isolated from any external environment, > like some offline industry automation machines, but then arguably > their running conditions would also be quite static and require very > little memory re-allocation. Vendor just decided not to included such hardware. That is why From ION to DMA-heap, people like to allocate from a cavout out memory. > > I also don't buy the performance reason. CPUs have been behind MMUs > for ages and nobody is running them with paging disabled for > performance reasons. Similarly, most of the modern consumer systems Page lookup would increase the delay. Besides a few upstream devices prove them only use a level 1 page table without TBL. > (mobile phones, PCs) run with IOMMUs enabled for pretty much anything > because of the security reason and they don't seem to be having any If the page is secure, you can't operate it in a insecure IOMMU or MMU. The most security way here, we should use a dedicated memory(or a zone in unified memory). I believe there are more users in mobile for DMA-heap than kernel's dma allocation API. > performance issues. In fact, it can improve the performance, because > memory allocation is much easier and without contiguous careouts (as > we used to have long ago on Android devices) the extra memory can be > used for buffers and caches to improve system performance. > > Best regards, > Tomasz > >> >>> >>> Best regards, >>> Tomasz >>> >>>>>>> Remove it from the core definitions but keep it for drivers internal >>>>>>> needs. >>>>>>> >>>>>>> Signed-off-by: Benjamin Gaignard <benjamin.gaignard@collabora.com> >>>>>>> --- >>>>>>> drivers/media/common/videobuf2/videobuf2-core.c | 2 ++ >>>>>>> drivers/media/platform/amphion/vdec.c | 1 + >>>>>>> .../media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c | 2 ++ >>>>>>> drivers/media/platform/qcom/venus/hfi.h | 2 ++ >>>>>>> drivers/media/platform/verisilicon/hantro_hw.h | 2 ++ >>>>>>> drivers/staging/media/ipu3/ipu3-v4l2.c | 2 ++ >>>>>>> include/media/videobuf2-core.h | 1 - >>>>>>> include/media/videobuf2-v4l2.h | 4 ---- >>>>>>> 8 files changed, 11 insertions(+), 5 deletions(-) >>>>>>> >>>>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>> b/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>> index 86e1e926fa45..899783f67580 100644 >>>>>>> --- a/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>> +++ b/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>> @@ -31,6 +31,8 @@ >>>>>>> >>>>>>> #include <trace/events/vb2.h> >>>>>>> >>>>>>> +#define VB2_MAX_FRAME 32 >>>>>>> + >>>>>>> static int debug; >>>>>>> module_param(debug, int, 0644); >>>>>>> >>>>>>> diff --git a/drivers/media/platform/amphion/vdec.c >>>>>>> b/drivers/media/platform/amphion/vdec.c >>>>>>> index 3fa1a74a2e20..b3219f6d17fa 100644 >>>>>>> --- a/drivers/media/platform/amphion/vdec.c >>>>>>> +++ b/drivers/media/platform/amphion/vdec.c >>>>>>> @@ -28,6 +28,7 @@ >>>>>>> >>>>>>> #define VDEC_MIN_BUFFER_CAP 8 >>>>>>> #define VDEC_MIN_BUFFER_OUT 8 >>>>>>> +#define VB2_MAX_FRAME 32 >>>>>>> >>>>>>> struct vdec_fs_info { >>>>>>> char name[8]; >>>>>>> diff --git >>>>>>> a/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >>>>>>> b/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >>>>>>> index 6532a69f1fa8..a1e0f24bb91c 100644 >>>>>>> --- a/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >>>>>>> +++ b/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >>>>>>> @@ -16,6 +16,8 @@ >>>>>>> #include "../vdec_drv_if.h" >>>>>>> #include "../vdec_vpu_if.h" >>>>>>> >>>>>>> +#define VB2_MAX_FRAME 32 >>>>>>> + >>>>>>> /* reset_frame_context defined in VP9 spec */ >>>>>>> #define VP9_RESET_FRAME_CONTEXT_NONE0 0 >>>>>>> #define VP9_RESET_FRAME_CONTEXT_NONE1 1 >>>>>>> diff --git a/drivers/media/platform/qcom/venus/hfi.h >>>>>>> b/drivers/media/platform/qcom/venus/hfi.h >>>>>>> index f25d412d6553..bd5ca5a8b945 100644 >>>>>>> --- a/drivers/media/platform/qcom/venus/hfi.h >>>>>>> +++ b/drivers/media/platform/qcom/venus/hfi.h >>>>>>> @@ -10,6 +10,8 @@ >>>>>>> >>>>>>> #include "hfi_helper.h" >>>>>>> >>>>>>> +#define VB2_MAX_FRAME 32 >>>>>>> + >>>>>>> #define VIDC_SESSION_TYPE_VPE 0 >>>>>>> #define VIDC_SESSION_TYPE_ENC 1 >>>>>>> #define VIDC_SESSION_TYPE_DEC 2 >>>>>>> diff --git a/drivers/media/platform/verisilicon/hantro_hw.h >>>>>>> b/drivers/media/platform/verisilicon/hantro_hw.h >>>>>>> index e83f0c523a30..9e8faf7ba6fb 100644 >>>>>>> --- a/drivers/media/platform/verisilicon/hantro_hw.h >>>>>>> +++ b/drivers/media/platform/verisilicon/hantro_hw.h >>>>>>> @@ -15,6 +15,8 @@ >>>>>>> #include <media/v4l2-vp9.h> >>>>>>> #include <media/videobuf2-core.h> >>>>>>> >>>>>>> +#define VB2_MAX_FRAME 32 >>>>>>> + >>>>>>> #define DEC_8190_ALIGN_MASK 0x07U >>>>>>> >>>>>>> #define MB_DIM 16 >>>>>>> diff --git a/drivers/staging/media/ipu3/ipu3-v4l2.c >>>>>>> b/drivers/staging/media/ipu3/ipu3-v4l2.c >>>>>>> index e530767e80a5..6627b5c2d4d6 100644 >>>>>>> --- a/drivers/staging/media/ipu3/ipu3-v4l2.c >>>>>>> +++ b/drivers/staging/media/ipu3/ipu3-v4l2.c >>>>>>> @@ -10,6 +10,8 @@ >>>>>>> #include "ipu3.h" >>>>>>> #include "ipu3-dmamap.h" >>>>>>> >>>>>>> +#define VB2_MAX_FRAME 32 >>>>>>> + >>>>>>> /******************** v4l2_subdev_ops ********************/ >>>>>>> >>>>>>> #define IPU3_RUNNING_MODE_VIDEO 0 >>>>>>> diff --git a/include/media/videobuf2-core.h >>>>>>> b/include/media/videobuf2-core.h >>>>>>> index 77921cf894ef..080b783d608d 100644 >>>>>>> --- a/include/media/videobuf2-core.h >>>>>>> +++ b/include/media/videobuf2-core.h >>>>>>> @@ -20,7 +20,6 @@ >>>>>>> #include <media/media-request.h> >>>>>>> #include <media/frame_vector.h> >>>>>>> >>>>>>> -#define VB2_MAX_FRAME (32) >>>>>>> #define VB2_MAX_PLANES (8) >>>>>>> >>>>>>> /** >>>>>>> diff --git a/include/media/videobuf2-v4l2.h >>>>>>> b/include/media/videobuf2-v4l2.h >>>>>>> index 5a845887850b..88a7a565170e 100644 >>>>>>> --- a/include/media/videobuf2-v4l2.h >>>>>>> +++ b/include/media/videobuf2-v4l2.h >>>>>>> @@ -15,10 +15,6 @@ >>>>>>> #include <linux/videodev2.h> >>>>>>> #include <media/videobuf2-core.h> >>>>>>> >>>>>>> -#if VB2_MAX_FRAME != VIDEO_MAX_FRAME >>>>>>> -#error VB2_MAX_FRAME != VIDEO_MAX_FRAME >>>>>>> -#endif >>>>>>> - >>>>>>> #if VB2_MAX_PLANES != VIDEO_MAX_PLANES >>>>>>> #error VB2_MAX_PLANES != VIDEO_MAX_PLANES >>>>>>> #endif >>>>>>> -- >>>>>>> 2.39.2 >>>>>>> >>>> -- >>>> Hsia-Jun(Randy) Li >>>> >> -- >> Hsia-Jun(Randy) Li >> -- Hsia-Jun(Randy) Li ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v3 04/11] media: videobuf2: Stop define VB2_MAX_FRAME as global 2023-07-28 6:55 ` Hsia-Jun Li @ 2023-07-28 7:03 ` Tomasz Figa 2023-07-28 7:24 ` Hsia-Jun Li 0 siblings, 1 reply; 27+ messages in thread From: Tomasz Figa @ 2023-07-28 7:03 UTC (permalink / raw) To: Hsia-Jun Li Cc: Benjamin Gaignard, linux-media, linux-kernel, linux-arm-kernel, m.szyprowski, linux-mediatek, linux-arm-msm, hverkuil-cisco, ezequiel, p.zabel, linux-rockchip, mchehab, linux-staging, ming.qian, kernel, gregkh, nicolas.dufresne On Fri, Jul 28, 2023 at 3:55 PM Hsia-Jun Li <Randy.Li@synaptics.com> wrote: > > > > On 7/28/23 14:46, Tomasz Figa wrote: > > CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe. > > > > > > On Mon, Jul 17, 2023 at 4:44 PM Hsia-Jun Li <Randy.Li@synaptics.com> wrote: > >> > >> > >> On 7/12/23 18:48, Tomasz Figa wrote: > >>> CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe. > >>> > >>> > >>> On Mon, Jul 03, 2023 at 04:35:30PM +0800, Hsia-Jun Li wrote: > >>>> On 7/3/23 16:09, Benjamin Gaignard wrote: > >>>>> CAUTION: Email originated externally, do not click links or open > >>>>> attachments unless you recognize the sender and know the content is > >>>>> safe. > >>>>> > >>>>> > >>>>> Le 30/06/2023 à 11:51, Hsia-Jun Li a écrit : > >>>>>> On 6/22/23 21:13, Benjamin Gaignard wrote: > >>>>>>> CAUTION: Email originated externally, do not click links or open > >>>>>>> attachments unless you recognize the sender and know the content is > >>>>>>> safe. > >>>>>>> > >>>>>>> > >>>>>>> After changing bufs arrays to a dynamic allocated array > >>>>>>> VB2_MAX_FRAME doesn't mean anything for videobuf2 core. > >>>>>> I think make it 64 which is the VB2_MAX_FRAME in Android GKI kernel is > >>>>>> more reasonable. > >>>>>> > >>>>>> It would be hard to iterate the whole array, it would go worse with a > >>>>>> filter. Such iterate may need to go twice because you mix > >>>>>> post-processing buffer and decoding buffer(with MV) in the same array. > >>>>> Here I don't want to change drivers behavior so I keep the same value. > >>>>> If it happens that they need more buffers, like for dynamic resolution > >>>>> change > >>>>> feature for Verisilicon VP9 decoder, case by case patches will be needed. > >>>>> > >>>> I just don't like the idea that using a variant length array here. > >>>> > >>> "I don't like" is not an argument. We had a number of arguments for > >>> using a generic helper (originally idr, but later decided to go with > >>> XArray, because the former is now deprecated) that we pointed out in > >>> our review comments for previous revisions. It wasn't really about the > >>> size being variable, but rather avoiding open coding things in vb2 and > >>> duplicating what's already implemented in generic code. > >> > >> I just want to say I don't think we need a variable length array to > >> store the buffer here. > >> > >> And the below is the reason that such a case could be avoided in the > >> first place. > >> > >>> > >>>> And I could explain why you won't need so many buffers for the performance > >>>> of decoding. > >>>> > >>>> VP9 could support 10 reference frames in dpb. > >>>> > >>>> Even for those frequent resolution changing test set, it would only happen > >>>> to two resolutions, > >>>> > >>>> 32 would be enough for 20 buffers of two resolution plus golden frames. It > >>>> also leaves enough slots for re-order latency. > >>>> > >>>> If your case had more two resolutions, likes low->medium->high. > >>>> > >>>> I would suggest just skip the medium resolutions, just allocate the lower > >>>> one first for fast playback then the highest for all the possible > >>>> > >>>> medium cases. Reallocation happens frequently would only cause memory > >>>> fragment, nothing benefits your performance. > >>>> > >>> We have mechanisms in the kernel to deal with memory fragmentation > >>> (migration/compaction) and it would still only matters for the > >>> pathologic cases of hardware that require physically contiguous memory. > >>> Modern hardware with proper DMA capabilities can either scatter-gather > >>> or are equipped with an IOMMU, so the allocation always happens in page > >>> granularity and fragmentation is avoided. > >> > >> Unfortunately, there are more devices that didn't have a IOMMU attached > >> to it, supporting scatter gather is more odd. > >> > >> It would be more likely that IOMMU would be disabled for the performance > >> reason. > > > > These days IOMMU is totally mandatory if you want to think about > > having any level of security in your system. Sure, there could be some > > systems that are completely isolated from any external environment, > > like some offline industry automation machines, but then arguably > > their running conditions would also be quite static and require very > > little memory re-allocation. > Vendor just decided not to included such hardware. > That is why From ION to DMA-heap, people like to allocate from a cavout > out memory. > > > > I also don't buy the performance reason. CPUs have been behind MMUs > > for ages and nobody is running them with paging disabled for > > performance reasons. Similarly, most of the modern consumer systems > Page lookup would increase the delay. Besides a few upstream devices > prove them only use a level 1 page table without TBL. That's just an excuse for a bad hardware design/implementation. As I said, there are good IOMMU implementations out there that don't suffer from performance issues. > > (mobile phones, PCs) run with IOMMUs enabled for pretty much anything > > because of the security reason and they don't seem to be having any > If the page is secure, you can't operate it in a insecure IOMMU or MMU. > The most security way here, we should use a dedicated memory(or a zone > in unified memory). You still need something to enforce that the hardware is not accessing memory that it's not supposed to access. How do you do that without an IOMMU? > I believe there are more users in mobile for DMA-heap than kernel's dma > allocation API. Yes, but that's completely separate from whether there is an IOMMU or not. It's just a different allocation API. > > performance issues. In fact, it can improve the performance, because > > memory allocation is much easier and without contiguous careouts (as > > we used to have long ago on Android devices) the extra memory can be > > used for buffers and caches to improve system performance. > > > > Best regards, > > Tomasz > > > >> > >>> > >>> Best regards, > >>> Tomasz > >>> > >>>>>>> Remove it from the core definitions but keep it for drivers internal > >>>>>>> needs. > >>>>>>> > >>>>>>> Signed-off-by: Benjamin Gaignard <benjamin.gaignard@collabora.com> > >>>>>>> --- > >>>>>>> drivers/media/common/videobuf2/videobuf2-core.c | 2 ++ > >>>>>>> drivers/media/platform/amphion/vdec.c | 1 + > >>>>>>> .../media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c | 2 ++ > >>>>>>> drivers/media/platform/qcom/venus/hfi.h | 2 ++ > >>>>>>> drivers/media/platform/verisilicon/hantro_hw.h | 2 ++ > >>>>>>> drivers/staging/media/ipu3/ipu3-v4l2.c | 2 ++ > >>>>>>> include/media/videobuf2-core.h | 1 - > >>>>>>> include/media/videobuf2-v4l2.h | 4 ---- > >>>>>>> 8 files changed, 11 insertions(+), 5 deletions(-) > >>>>>>> > >>>>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c > >>>>>>> b/drivers/media/common/videobuf2/videobuf2-core.c > >>>>>>> index 86e1e926fa45..899783f67580 100644 > >>>>>>> --- a/drivers/media/common/videobuf2/videobuf2-core.c > >>>>>>> +++ b/drivers/media/common/videobuf2/videobuf2-core.c > >>>>>>> @@ -31,6 +31,8 @@ > >>>>>>> > >>>>>>> #include <trace/events/vb2.h> > >>>>>>> > >>>>>>> +#define VB2_MAX_FRAME 32 > >>>>>>> + > >>>>>>> static int debug; > >>>>>>> module_param(debug, int, 0644); > >>>>>>> > >>>>>>> diff --git a/drivers/media/platform/amphion/vdec.c > >>>>>>> b/drivers/media/platform/amphion/vdec.c > >>>>>>> index 3fa1a74a2e20..b3219f6d17fa 100644 > >>>>>>> --- a/drivers/media/platform/amphion/vdec.c > >>>>>>> +++ b/drivers/media/platform/amphion/vdec.c > >>>>>>> @@ -28,6 +28,7 @@ > >>>>>>> > >>>>>>> #define VDEC_MIN_BUFFER_CAP 8 > >>>>>>> #define VDEC_MIN_BUFFER_OUT 8 > >>>>>>> +#define VB2_MAX_FRAME 32 > >>>>>>> > >>>>>>> struct vdec_fs_info { > >>>>>>> char name[8]; > >>>>>>> diff --git > >>>>>>> a/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c > >>>>>>> b/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c > >>>>>>> index 6532a69f1fa8..a1e0f24bb91c 100644 > >>>>>>> --- a/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c > >>>>>>> +++ b/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c > >>>>>>> @@ -16,6 +16,8 @@ > >>>>>>> #include "../vdec_drv_if.h" > >>>>>>> #include "../vdec_vpu_if.h" > >>>>>>> > >>>>>>> +#define VB2_MAX_FRAME 32 > >>>>>>> + > >>>>>>> /* reset_frame_context defined in VP9 spec */ > >>>>>>> #define VP9_RESET_FRAME_CONTEXT_NONE0 0 > >>>>>>> #define VP9_RESET_FRAME_CONTEXT_NONE1 1 > >>>>>>> diff --git a/drivers/media/platform/qcom/venus/hfi.h > >>>>>>> b/drivers/media/platform/qcom/venus/hfi.h > >>>>>>> index f25d412d6553..bd5ca5a8b945 100644 > >>>>>>> --- a/drivers/media/platform/qcom/venus/hfi.h > >>>>>>> +++ b/drivers/media/platform/qcom/venus/hfi.h > >>>>>>> @@ -10,6 +10,8 @@ > >>>>>>> > >>>>>>> #include "hfi_helper.h" > >>>>>>> > >>>>>>> +#define VB2_MAX_FRAME 32 > >>>>>>> + > >>>>>>> #define VIDC_SESSION_TYPE_VPE 0 > >>>>>>> #define VIDC_SESSION_TYPE_ENC 1 > >>>>>>> #define VIDC_SESSION_TYPE_DEC 2 > >>>>>>> diff --git a/drivers/media/platform/verisilicon/hantro_hw.h > >>>>>>> b/drivers/media/platform/verisilicon/hantro_hw.h > >>>>>>> index e83f0c523a30..9e8faf7ba6fb 100644 > >>>>>>> --- a/drivers/media/platform/verisilicon/hantro_hw.h > >>>>>>> +++ b/drivers/media/platform/verisilicon/hantro_hw.h > >>>>>>> @@ -15,6 +15,8 @@ > >>>>>>> #include <media/v4l2-vp9.h> > >>>>>>> #include <media/videobuf2-core.h> > >>>>>>> > >>>>>>> +#define VB2_MAX_FRAME 32 > >>>>>>> + > >>>>>>> #define DEC_8190_ALIGN_MASK 0x07U > >>>>>>> > >>>>>>> #define MB_DIM 16 > >>>>>>> diff --git a/drivers/staging/media/ipu3/ipu3-v4l2.c > >>>>>>> b/drivers/staging/media/ipu3/ipu3-v4l2.c > >>>>>>> index e530767e80a5..6627b5c2d4d6 100644 > >>>>>>> --- a/drivers/staging/media/ipu3/ipu3-v4l2.c > >>>>>>> +++ b/drivers/staging/media/ipu3/ipu3-v4l2.c > >>>>>>> @@ -10,6 +10,8 @@ > >>>>>>> #include "ipu3.h" > >>>>>>> #include "ipu3-dmamap.h" > >>>>>>> > >>>>>>> +#define VB2_MAX_FRAME 32 > >>>>>>> + > >>>>>>> /******************** v4l2_subdev_ops ********************/ > >>>>>>> > >>>>>>> #define IPU3_RUNNING_MODE_VIDEO 0 > >>>>>>> diff --git a/include/media/videobuf2-core.h > >>>>>>> b/include/media/videobuf2-core.h > >>>>>>> index 77921cf894ef..080b783d608d 100644 > >>>>>>> --- a/include/media/videobuf2-core.h > >>>>>>> +++ b/include/media/videobuf2-core.h > >>>>>>> @@ -20,7 +20,6 @@ > >>>>>>> #include <media/media-request.h> > >>>>>>> #include <media/frame_vector.h> > >>>>>>> > >>>>>>> -#define VB2_MAX_FRAME (32) > >>>>>>> #define VB2_MAX_PLANES (8) > >>>>>>> > >>>>>>> /** > >>>>>>> diff --git a/include/media/videobuf2-v4l2.h > >>>>>>> b/include/media/videobuf2-v4l2.h > >>>>>>> index 5a845887850b..88a7a565170e 100644 > >>>>>>> --- a/include/media/videobuf2-v4l2.h > >>>>>>> +++ b/include/media/videobuf2-v4l2.h > >>>>>>> @@ -15,10 +15,6 @@ > >>>>>>> #include <linux/videodev2.h> > >>>>>>> #include <media/videobuf2-core.h> > >>>>>>> > >>>>>>> -#if VB2_MAX_FRAME != VIDEO_MAX_FRAME > >>>>>>> -#error VB2_MAX_FRAME != VIDEO_MAX_FRAME > >>>>>>> -#endif > >>>>>>> - > >>>>>>> #if VB2_MAX_PLANES != VIDEO_MAX_PLANES > >>>>>>> #error VB2_MAX_PLANES != VIDEO_MAX_PLANES > >>>>>>> #endif > >>>>>>> -- > >>>>>>> 2.39.2 > >>>>>>> > >>>> -- > >>>> Hsia-Jun(Randy) Li > >>>> > >> -- > >> Hsia-Jun(Randy) Li > >> > > -- > Hsia-Jun(Randy) Li ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v3 04/11] media: videobuf2: Stop define VB2_MAX_FRAME as global 2023-07-28 7:03 ` Tomasz Figa @ 2023-07-28 7:24 ` Hsia-Jun Li 2023-09-07 4:15 ` Tomasz Figa 0 siblings, 1 reply; 27+ messages in thread From: Hsia-Jun Li @ 2023-07-28 7:24 UTC (permalink / raw) To: Tomasz Figa Cc: Benjamin Gaignard, linux-media, linux-kernel, linux-arm-kernel, m.szyprowski, linux-mediatek, linux-arm-msm, hverkuil-cisco, ezequiel, p.zabel, linux-rockchip, mchehab, linux-staging, ming.qian, kernel, gregkh, nicolas.dufresne On 7/28/23 15:03, Tomasz Figa wrote: > CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe. > > > On Fri, Jul 28, 2023 at 3:55 PM Hsia-Jun Li <Randy.Li@synaptics.com> wrote: >> >> >> >> On 7/28/23 14:46, Tomasz Figa wrote: >>> CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe. >>> >>> >>> On Mon, Jul 17, 2023 at 4:44 PM Hsia-Jun Li <Randy.Li@synaptics.com> wrote: >>>> >>>> >>>> On 7/12/23 18:48, Tomasz Figa wrote: >>>>> CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe. >>>>> >>>>> >>>>> On Mon, Jul 03, 2023 at 04:35:30PM +0800, Hsia-Jun Li wrote: >>>>>> On 7/3/23 16:09, Benjamin Gaignard wrote: >>>>>>> CAUTION: Email originated externally, do not click links or open >>>>>>> attachments unless you recognize the sender and know the content is >>>>>>> safe. >>>>>>> >>>>>>> >>>>>>> Le 30/06/2023 à 11:51, Hsia-Jun Li a écrit : >>>>>>>> On 6/22/23 21:13, Benjamin Gaignard wrote: >>>>>>>>> CAUTION: Email originated externally, do not click links or open >>>>>>>>> attachments unless you recognize the sender and know the content is >>>>>>>>> safe. >>>>>>>>> >>>>>>>>> >>>>>>>>> After changing bufs arrays to a dynamic allocated array >>>>>>>>> VB2_MAX_FRAME doesn't mean anything for videobuf2 core. >>>>>>>> I think make it 64 which is the VB2_MAX_FRAME in Android GKI kernel is >>>>>>>> more reasonable. >>>>>>>> >>>>>>>> It would be hard to iterate the whole array, it would go worse with a >>>>>>>> filter. Such iterate may need to go twice because you mix >>>>>>>> post-processing buffer and decoding buffer(with MV) in the same array. >>>>>>> Here I don't want to change drivers behavior so I keep the same value. >>>>>>> If it happens that they need more buffers, like for dynamic resolution >>>>>>> change >>>>>>> feature for Verisilicon VP9 decoder, case by case patches will be needed. >>>>>>> >>>>>> I just don't like the idea that using a variant length array here. >>>>>> >>>>> "I don't like" is not an argument. We had a number of arguments for >>>>> using a generic helper (originally idr, but later decided to go with >>>>> XArray, because the former is now deprecated) that we pointed out in >>>>> our review comments for previous revisions. It wasn't really about the >>>>> size being variable, but rather avoiding open coding things in vb2 and >>>>> duplicating what's already implemented in generic code. >>>> >>>> I just want to say I don't think we need a variable length array to >>>> store the buffer here. >>>> >>>> And the below is the reason that such a case could be avoided in the >>>> first place. >>>> >>>>> >>>>>> And I could explain why you won't need so many buffers for the performance >>>>>> of decoding. >>>>>> >>>>>> VP9 could support 10 reference frames in dpb. >>>>>> >>>>>> Even for those frequent resolution changing test set, it would only happen >>>>>> to two resolutions, >>>>>> >>>>>> 32 would be enough for 20 buffers of two resolution plus golden frames. It >>>>>> also leaves enough slots for re-order latency. >>>>>> >>>>>> If your case had more two resolutions, likes low->medium->high. >>>>>> >>>>>> I would suggest just skip the medium resolutions, just allocate the lower >>>>>> one first for fast playback then the highest for all the possible >>>>>> >>>>>> medium cases. Reallocation happens frequently would only cause memory >>>>>> fragment, nothing benefits your performance. >>>>>> >>>>> We have mechanisms in the kernel to deal with memory fragmentation >>>>> (migration/compaction) and it would still only matters for the >>>>> pathologic cases of hardware that require physically contiguous memory. >>>>> Modern hardware with proper DMA capabilities can either scatter-gather >>>>> or are equipped with an IOMMU, so the allocation always happens in page >>>>> granularity and fragmentation is avoided. >>>> >>>> Unfortunately, there are more devices that didn't have a IOMMU attached >>>> to it, supporting scatter gather is more odd. >>>> >>>> It would be more likely that IOMMU would be disabled for the performance >>>> reason. >>> >>> These days IOMMU is totally mandatory if you want to think about >>> having any level of security in your system. Sure, there could be some >>> systems that are completely isolated from any external environment, >>> like some offline industry automation machines, but then arguably >>> their running conditions would also be quite static and require very >>> little memory re-allocation. >> Vendor just decided not to included such hardware. >> That is why From ION to DMA-heap, people like to allocate from a cavout >> out memory. >>> >>> I also don't buy the performance reason. CPUs have been behind MMUs >>> for ages and nobody is running them with paging disabled for >>> performance reasons. Similarly, most of the modern consumer systems >> Page lookup would increase the delay. Besides a few upstream devices >> prove them only use a level 1 page table without TBL. > > That's just an excuse for a bad hardware design/implementation. As I > said, there are good IOMMU implementations out there that don't suffer > from performance issues. > I could do nothing about that. Besides, even with TLB, cache missing could happen frequently, especially we need to access many (5~16, 10 usually) buffers and more 11MBytes each in a hardware processing. You can't have a very large TLB. >>> (mobile phones, PCs) run with IOMMUs enabled for pretty much anything >>> because of the security reason and they don't seem to be having any >> If the page is secure, you can't operate it in a insecure IOMMU or MMU. >> The most security way here, we should use a dedicated memory(or a zone >> in unified memory). > > You still need something to enforce that the hardware is not accessing > memory that it's not supposed to access. How do you do that without an > IOMMU? > If you know the arm security pipeline and security controller, you could found we could reserved a range of memory for a security id(devices in secure world may be a different security domain). Besides, a MMU or security MPU could mark some pages for the secure world access only, it doesn't mean the device need an IOMMU to access them. The MPU could filter the access through the AXI id. >> I believe there are more users in mobile for DMA-heap than kernel's dma >> allocation API. > > Yes, but that's completely separate from whether there is an IOMMU or > not. It's just a different allocation API. > The memory heap would mean a dedicated memory usually(we don't talk about system heap or why there are many vendor heaps). Dedicated memory means contiguous memory in the most of cases. >>> performance issues. In fact, it can improve the performance, because >>> memory allocation is much easier and without contiguous careouts (as >>> we used to have long ago on Android devices) the extra memory can be >>> used for buffers and caches to improve system performance. >>> >>> Best regards, >>> Tomasz >>> >>>> >>>>> >>>>> Best regards, >>>>> Tomasz >>>>> >>>>>>>>> Remove it from the core definitions but keep it for drivers internal >>>>>>>>> needs. >>>>>>>>> >>>>>>>>> Signed-off-by: Benjamin Gaignard <benjamin.gaignard@collabora.com> >>>>>>>>> --- >>>>>>>>> drivers/media/common/videobuf2/videobuf2-core.c | 2 ++ >>>>>>>>> drivers/media/platform/amphion/vdec.c | 1 + >>>>>>>>> .../media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c | 2 ++ >>>>>>>>> drivers/media/platform/qcom/venus/hfi.h | 2 ++ >>>>>>>>> drivers/media/platform/verisilicon/hantro_hw.h | 2 ++ >>>>>>>>> drivers/staging/media/ipu3/ipu3-v4l2.c | 2 ++ >>>>>>>>> include/media/videobuf2-core.h | 1 - >>>>>>>>> include/media/videobuf2-v4l2.h | 4 ---- >>>>>>>>> 8 files changed, 11 insertions(+), 5 deletions(-) >>>>>>>>> >>>>>>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>>> b/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>>> index 86e1e926fa45..899783f67580 100644 >>>>>>>>> --- a/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>>> +++ b/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>>> @@ -31,6 +31,8 @@ >>>>>>>>> >>>>>>>>> #include <trace/events/vb2.h> >>>>>>>>> >>>>>>>>> +#define VB2_MAX_FRAME 32 >>>>>>>>> + >>>>>>>>> static int debug; >>>>>>>>> module_param(debug, int, 0644); >>>>>>>>> >>>>>>>>> diff --git a/drivers/media/platform/amphion/vdec.c >>>>>>>>> b/drivers/media/platform/amphion/vdec.c >>>>>>>>> index 3fa1a74a2e20..b3219f6d17fa 100644 >>>>>>>>> --- a/drivers/media/platform/amphion/vdec.c >>>>>>>>> +++ b/drivers/media/platform/amphion/vdec.c >>>>>>>>> @@ -28,6 +28,7 @@ >>>>>>>>> >>>>>>>>> #define VDEC_MIN_BUFFER_CAP 8 >>>>>>>>> #define VDEC_MIN_BUFFER_OUT 8 >>>>>>>>> +#define VB2_MAX_FRAME 32 >>>>>>>>> >>>>>>>>> struct vdec_fs_info { >>>>>>>>> char name[8]; >>>>>>>>> diff --git >>>>>>>>> a/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >>>>>>>>> b/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >>>>>>>>> index 6532a69f1fa8..a1e0f24bb91c 100644 >>>>>>>>> --- a/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >>>>>>>>> +++ b/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >>>>>>>>> @@ -16,6 +16,8 @@ >>>>>>>>> #include "../vdec_drv_if.h" >>>>>>>>> #include "../vdec_vpu_if.h" >>>>>>>>> >>>>>>>>> +#define VB2_MAX_FRAME 32 >>>>>>>>> + >>>>>>>>> /* reset_frame_context defined in VP9 spec */ >>>>>>>>> #define VP9_RESET_FRAME_CONTEXT_NONE0 0 >>>>>>>>> #define VP9_RESET_FRAME_CONTEXT_NONE1 1 >>>>>>>>> diff --git a/drivers/media/platform/qcom/venus/hfi.h >>>>>>>>> b/drivers/media/platform/qcom/venus/hfi.h >>>>>>>>> index f25d412d6553..bd5ca5a8b945 100644 >>>>>>>>> --- a/drivers/media/platform/qcom/venus/hfi.h >>>>>>>>> +++ b/drivers/media/platform/qcom/venus/hfi.h >>>>>>>>> @@ -10,6 +10,8 @@ >>>>>>>>> >>>>>>>>> #include "hfi_helper.h" >>>>>>>>> >>>>>>>>> +#define VB2_MAX_FRAME 32 >>>>>>>>> + >>>>>>>>> #define VIDC_SESSION_TYPE_VPE 0 >>>>>>>>> #define VIDC_SESSION_TYPE_ENC 1 >>>>>>>>> #define VIDC_SESSION_TYPE_DEC 2 >>>>>>>>> diff --git a/drivers/media/platform/verisilicon/hantro_hw.h >>>>>>>>> b/drivers/media/platform/verisilicon/hantro_hw.h >>>>>>>>> index e83f0c523a30..9e8faf7ba6fb 100644 >>>>>>>>> --- a/drivers/media/platform/verisilicon/hantro_hw.h >>>>>>>>> +++ b/drivers/media/platform/verisilicon/hantro_hw.h >>>>>>>>> @@ -15,6 +15,8 @@ >>>>>>>>> #include <media/v4l2-vp9.h> >>>>>>>>> #include <media/videobuf2-core.h> >>>>>>>>> >>>>>>>>> +#define VB2_MAX_FRAME 32 >>>>>>>>> + >>>>>>>>> #define DEC_8190_ALIGN_MASK 0x07U >>>>>>>>> >>>>>>>>> #define MB_DIM 16 >>>>>>>>> diff --git a/drivers/staging/media/ipu3/ipu3-v4l2.c >>>>>>>>> b/drivers/staging/media/ipu3/ipu3-v4l2.c >>>>>>>>> index e530767e80a5..6627b5c2d4d6 100644 >>>>>>>>> --- a/drivers/staging/media/ipu3/ipu3-v4l2.c >>>>>>>>> +++ b/drivers/staging/media/ipu3/ipu3-v4l2.c >>>>>>>>> @@ -10,6 +10,8 @@ >>>>>>>>> #include "ipu3.h" >>>>>>>>> #include "ipu3-dmamap.h" >>>>>>>>> >>>>>>>>> +#define VB2_MAX_FRAME 32 >>>>>>>>> + >>>>>>>>> /******************** v4l2_subdev_ops ********************/ >>>>>>>>> >>>>>>>>> #define IPU3_RUNNING_MODE_VIDEO 0 >>>>>>>>> diff --git a/include/media/videobuf2-core.h >>>>>>>>> b/include/media/videobuf2-core.h >>>>>>>>> index 77921cf894ef..080b783d608d 100644 >>>>>>>>> --- a/include/media/videobuf2-core.h >>>>>>>>> +++ b/include/media/videobuf2-core.h >>>>>>>>> @@ -20,7 +20,6 @@ >>>>>>>>> #include <media/media-request.h> >>>>>>>>> #include <media/frame_vector.h> >>>>>>>>> >>>>>>>>> -#define VB2_MAX_FRAME (32) >>>>>>>>> #define VB2_MAX_PLANES (8) >>>>>>>>> >>>>>>>>> /** >>>>>>>>> diff --git a/include/media/videobuf2-v4l2.h >>>>>>>>> b/include/media/videobuf2-v4l2.h >>>>>>>>> index 5a845887850b..88a7a565170e 100644 >>>>>>>>> --- a/include/media/videobuf2-v4l2.h >>>>>>>>> +++ b/include/media/videobuf2-v4l2.h >>>>>>>>> @@ -15,10 +15,6 @@ >>>>>>>>> #include <linux/videodev2.h> >>>>>>>>> #include <media/videobuf2-core.h> >>>>>>>>> >>>>>>>>> -#if VB2_MAX_FRAME != VIDEO_MAX_FRAME >>>>>>>>> -#error VB2_MAX_FRAME != VIDEO_MAX_FRAME >>>>>>>>> -#endif >>>>>>>>> - >>>>>>>>> #if VB2_MAX_PLANES != VIDEO_MAX_PLANES >>>>>>>>> #error VB2_MAX_PLANES != VIDEO_MAX_PLANES >>>>>>>>> #endif >>>>>>>>> -- >>>>>>>>> 2.39.2 >>>>>>>>> >>>>>> -- >>>>>> Hsia-Jun(Randy) Li >>>>>> >>>> -- >>>> Hsia-Jun(Randy) Li >>>> >> >> -- >> Hsia-Jun(Randy) Li -- Hsia-Jun(Randy) Li ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v3 04/11] media: videobuf2: Stop define VB2_MAX_FRAME as global 2023-07-28 7:24 ` Hsia-Jun Li @ 2023-09-07 4:15 ` Tomasz Figa 2023-09-07 6:54 ` Hsia-Jun Li 0 siblings, 1 reply; 27+ messages in thread From: Tomasz Figa @ 2023-09-07 4:15 UTC (permalink / raw) To: Hsia-Jun Li Cc: Benjamin Gaignard, linux-media, linux-kernel, linux-arm-kernel, m.szyprowski, linux-mediatek, linux-arm-msm, hverkuil-cisco, ezequiel, p.zabel, linux-rockchip, mchehab, linux-staging, ming.qian, kernel, gregkh, nicolas.dufresne On Fri, Jul 28, 2023 at 4:25 PM Hsia-Jun Li <Randy.Li@synaptics.com> wrote: > > > > On 7/28/23 15:03, Tomasz Figa wrote: > > CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe. > > > > > > On Fri, Jul 28, 2023 at 3:55 PM Hsia-Jun Li <Randy.Li@synaptics.com> wrote: > >> > >> > >> > >> On 7/28/23 14:46, Tomasz Figa wrote: > >>> CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe. > >>> > >>> > >>> On Mon, Jul 17, 2023 at 4:44 PM Hsia-Jun Li <Randy.Li@synaptics.com> wrote: > >>>> > >>>> > >>>> On 7/12/23 18:48, Tomasz Figa wrote: > >>>>> CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe. > >>>>> > >>>>> > >>>>> On Mon, Jul 03, 2023 at 04:35:30PM +0800, Hsia-Jun Li wrote: > >>>>>> On 7/3/23 16:09, Benjamin Gaignard wrote: > >>>>>>> CAUTION: Email originated externally, do not click links or open > >>>>>>> attachments unless you recognize the sender and know the content is > >>>>>>> safe. > >>>>>>> > >>>>>>> > >>>>>>> Le 30/06/2023 à 11:51, Hsia-Jun Li a écrit : > >>>>>>>> On 6/22/23 21:13, Benjamin Gaignard wrote: > >>>>>>>>> CAUTION: Email originated externally, do not click links or open > >>>>>>>>> attachments unless you recognize the sender and know the content is > >>>>>>>>> safe. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> After changing bufs arrays to a dynamic allocated array > >>>>>>>>> VB2_MAX_FRAME doesn't mean anything for videobuf2 core. > >>>>>>>> I think make it 64 which is the VB2_MAX_FRAME in Android GKI kernel is > >>>>>>>> more reasonable. > >>>>>>>> > >>>>>>>> It would be hard to iterate the whole array, it would go worse with a > >>>>>>>> filter. Such iterate may need to go twice because you mix > >>>>>>>> post-processing buffer and decoding buffer(with MV) in the same array. > >>>>>>> Here I don't want to change drivers behavior so I keep the same value. > >>>>>>> If it happens that they need more buffers, like for dynamic resolution > >>>>>>> change > >>>>>>> feature for Verisilicon VP9 decoder, case by case patches will be needed. > >>>>>>> > >>>>>> I just don't like the idea that using a variant length array here. > >>>>>> > >>>>> "I don't like" is not an argument. We had a number of arguments for > >>>>> using a generic helper (originally idr, but later decided to go with > >>>>> XArray, because the former is now deprecated) that we pointed out in > >>>>> our review comments for previous revisions. It wasn't really about the > >>>>> size being variable, but rather avoiding open coding things in vb2 and > >>>>> duplicating what's already implemented in generic code. > >>>> > >>>> I just want to say I don't think we need a variable length array to > >>>> store the buffer here. > >>>> > >>>> And the below is the reason that such a case could be avoided in the > >>>> first place. > >>>> > >>>>> > >>>>>> And I could explain why you won't need so many buffers for the performance > >>>>>> of decoding. > >>>>>> > >>>>>> VP9 could support 10 reference frames in dpb. > >>>>>> > >>>>>> Even for those frequent resolution changing test set, it would only happen > >>>>>> to two resolutions, > >>>>>> > >>>>>> 32 would be enough for 20 buffers of two resolution plus golden frames. It > >>>>>> also leaves enough slots for re-order latency. > >>>>>> > >>>>>> If your case had more two resolutions, likes low->medium->high. > >>>>>> > >>>>>> I would suggest just skip the medium resolutions, just allocate the lower > >>>>>> one first for fast playback then the highest for all the possible > >>>>>> > >>>>>> medium cases. Reallocation happens frequently would only cause memory > >>>>>> fragment, nothing benefits your performance. > >>>>>> > >>>>> We have mechanisms in the kernel to deal with memory fragmentation > >>>>> (migration/compaction) and it would still only matters for the > >>>>> pathologic cases of hardware that require physically contiguous memory. > >>>>> Modern hardware with proper DMA capabilities can either scatter-gather > >>>>> or are equipped with an IOMMU, so the allocation always happens in page > >>>>> granularity and fragmentation is avoided. > >>>> > >>>> Unfortunately, there are more devices that didn't have a IOMMU attached > >>>> to it, supporting scatter gather is more odd. > >>>> > >>>> It would be more likely that IOMMU would be disabled for the performance > >>>> reason. > >>> > >>> These days IOMMU is totally mandatory if you want to think about > >>> having any level of security in your system. Sure, there could be some > >>> systems that are completely isolated from any external environment, > >>> like some offline industry automation machines, but then arguably > >>> their running conditions would also be quite static and require very > >>> little memory re-allocation. > >> Vendor just decided not to included such hardware. > >> That is why From ION to DMA-heap, people like to allocate from a cavout > >> out memory. > >>> > >>> I also don't buy the performance reason. CPUs have been behind MMUs > >>> for ages and nobody is running them with paging disabled for > >>> performance reasons. Similarly, most of the modern consumer systems > >> Page lookup would increase the delay. Besides a few upstream devices > >> prove them only use a level 1 page table without TBL. > > > > That's just an excuse for a bad hardware design/implementation. As I > > said, there are good IOMMU implementations out there that don't suffer > > from performance issues. > > > I could do nothing about that. > Besides, even with TLB, cache missing could happen frequently, > especially we need to access many (5~16, 10 usually) buffers and more > 11MBytes each in a hardware processing. > You can't have a very large TLB. Right, but as I wrote in my previous emails, we have the right methods in the kernel for providing drivers with contiguous memory and those can be used for those special cases. > >>> (mobile phones, PCs) run with IOMMUs enabled for pretty much anything > >>> because of the security reason and they don't seem to be having any > >> If the page is secure, you can't operate it in a insecure IOMMU or MMU. > >> The most security way here, we should use a dedicated memory(or a zone > >> in unified memory). > > > > You still need something to enforce that the hardware is not accessing > > memory that it's not supposed to access. How do you do that without an > > IOMMU? > > > If you know the arm security pipeline and security controller, you could > found we could reserved a range of memory for a security id(devices in > secure world may be a different security domain). > Besides, a MMU or security MPU could mark some pages for the secure > world access only, it doesn't mean the device need an IOMMU to access > them. The MPU could filter the access through the AXI id. > >> I believe there are more users in mobile for DMA-heap than kernel's dma > >> allocation API. > > > > Yes, but that's completely separate from whether there is an IOMMU or > > not. It's just a different allocation API. > > > The memory heap would mean a dedicated memory usually(we don't talk > about system heap or why there are many vendor heaps). Dedicated memory > means contiguous memory in the most of cases. No, and no. First no - DMA-buf heap doesn't imply dedicated memory and usually one wants to completely avoid carving out memory, because it becomes useless if specific use case is not active. Second no - there are ways to provide dedicated memory regions to the DMA mapping API, such as shared or restricted DMA pool [1]. [1] https://elixir.bootlin.com/linux/latest/source/Documentation/devicetree/bindings/reserved-memory/shared-dma-pool.yaml Best regards, Tomasz > >>> performance issues. In fact, it can improve the performance, because > >>> memory allocation is much easier and without contiguous careouts (as > >>> we used to have long ago on Android devices) the extra memory can be > >>> used for buffers and caches to improve system performance. > >>> > >>> Best regards, > >>> Tomasz > >>> > >>>> > >>>>> > >>>>> Best regards, > >>>>> Tomasz > >>>>> > >>>>>>>>> Remove it from the core definitions but keep it for drivers internal > >>>>>>>>> needs. > >>>>>>>>> > >>>>>>>>> Signed-off-by: Benjamin Gaignard <benjamin.gaignard@collabora.com> > >>>>>>>>> --- > >>>>>>>>> drivers/media/common/videobuf2/videobuf2-core.c | 2 ++ > >>>>>>>>> drivers/media/platform/amphion/vdec.c | 1 + > >>>>>>>>> .../media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c | 2 ++ > >>>>>>>>> drivers/media/platform/qcom/venus/hfi.h | 2 ++ > >>>>>>>>> drivers/media/platform/verisilicon/hantro_hw.h | 2 ++ > >>>>>>>>> drivers/staging/media/ipu3/ipu3-v4l2.c | 2 ++ > >>>>>>>>> include/media/videobuf2-core.h | 1 - > >>>>>>>>> include/media/videobuf2-v4l2.h | 4 ---- > >>>>>>>>> 8 files changed, 11 insertions(+), 5 deletions(-) > >>>>>>>>> > >>>>>>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c > >>>>>>>>> b/drivers/media/common/videobuf2/videobuf2-core.c > >>>>>>>>> index 86e1e926fa45..899783f67580 100644 > >>>>>>>>> --- a/drivers/media/common/videobuf2/videobuf2-core.c > >>>>>>>>> +++ b/drivers/media/common/videobuf2/videobuf2-core.c > >>>>>>>>> @@ -31,6 +31,8 @@ > >>>>>>>>> > >>>>>>>>> #include <trace/events/vb2.h> > >>>>>>>>> > >>>>>>>>> +#define VB2_MAX_FRAME 32 > >>>>>>>>> + > >>>>>>>>> static int debug; > >>>>>>>>> module_param(debug, int, 0644); > >>>>>>>>> > >>>>>>>>> diff --git a/drivers/media/platform/amphion/vdec.c > >>>>>>>>> b/drivers/media/platform/amphion/vdec.c > >>>>>>>>> index 3fa1a74a2e20..b3219f6d17fa 100644 > >>>>>>>>> --- a/drivers/media/platform/amphion/vdec.c > >>>>>>>>> +++ b/drivers/media/platform/amphion/vdec.c > >>>>>>>>> @@ -28,6 +28,7 @@ > >>>>>>>>> > >>>>>>>>> #define VDEC_MIN_BUFFER_CAP 8 > >>>>>>>>> #define VDEC_MIN_BUFFER_OUT 8 > >>>>>>>>> +#define VB2_MAX_FRAME 32 > >>>>>>>>> > >>>>>>>>> struct vdec_fs_info { > >>>>>>>>> char name[8]; > >>>>>>>>> diff --git > >>>>>>>>> a/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c > >>>>>>>>> b/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c > >>>>>>>>> index 6532a69f1fa8..a1e0f24bb91c 100644 > >>>>>>>>> --- a/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c > >>>>>>>>> +++ b/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c > >>>>>>>>> @@ -16,6 +16,8 @@ > >>>>>>>>> #include "../vdec_drv_if.h" > >>>>>>>>> #include "../vdec_vpu_if.h" > >>>>>>>>> > >>>>>>>>> +#define VB2_MAX_FRAME 32 > >>>>>>>>> + > >>>>>>>>> /* reset_frame_context defined in VP9 spec */ > >>>>>>>>> #define VP9_RESET_FRAME_CONTEXT_NONE0 0 > >>>>>>>>> #define VP9_RESET_FRAME_CONTEXT_NONE1 1 > >>>>>>>>> diff --git a/drivers/media/platform/qcom/venus/hfi.h > >>>>>>>>> b/drivers/media/platform/qcom/venus/hfi.h > >>>>>>>>> index f25d412d6553..bd5ca5a8b945 100644 > >>>>>>>>> --- a/drivers/media/platform/qcom/venus/hfi.h > >>>>>>>>> +++ b/drivers/media/platform/qcom/venus/hfi.h > >>>>>>>>> @@ -10,6 +10,8 @@ > >>>>>>>>> > >>>>>>>>> #include "hfi_helper.h" > >>>>>>>>> > >>>>>>>>> +#define VB2_MAX_FRAME 32 > >>>>>>>>> + > >>>>>>>>> #define VIDC_SESSION_TYPE_VPE 0 > >>>>>>>>> #define VIDC_SESSION_TYPE_ENC 1 > >>>>>>>>> #define VIDC_SESSION_TYPE_DEC 2 > >>>>>>>>> diff --git a/drivers/media/platform/verisilicon/hantro_hw.h > >>>>>>>>> b/drivers/media/platform/verisilicon/hantro_hw.h > >>>>>>>>> index e83f0c523a30..9e8faf7ba6fb 100644 > >>>>>>>>> --- a/drivers/media/platform/verisilicon/hantro_hw.h > >>>>>>>>> +++ b/drivers/media/platform/verisilicon/hantro_hw.h > >>>>>>>>> @@ -15,6 +15,8 @@ > >>>>>>>>> #include <media/v4l2-vp9.h> > >>>>>>>>> #include <media/videobuf2-core.h> > >>>>>>>>> > >>>>>>>>> +#define VB2_MAX_FRAME 32 > >>>>>>>>> + > >>>>>>>>> #define DEC_8190_ALIGN_MASK 0x07U > >>>>>>>>> > >>>>>>>>> #define MB_DIM 16 > >>>>>>>>> diff --git a/drivers/staging/media/ipu3/ipu3-v4l2.c > >>>>>>>>> b/drivers/staging/media/ipu3/ipu3-v4l2.c > >>>>>>>>> index e530767e80a5..6627b5c2d4d6 100644 > >>>>>>>>> --- a/drivers/staging/media/ipu3/ipu3-v4l2.c > >>>>>>>>> +++ b/drivers/staging/media/ipu3/ipu3-v4l2.c > >>>>>>>>> @@ -10,6 +10,8 @@ > >>>>>>>>> #include "ipu3.h" > >>>>>>>>> #include "ipu3-dmamap.h" > >>>>>>>>> > >>>>>>>>> +#define VB2_MAX_FRAME 32 > >>>>>>>>> + > >>>>>>>>> /******************** v4l2_subdev_ops ********************/ > >>>>>>>>> > >>>>>>>>> #define IPU3_RUNNING_MODE_VIDEO 0 > >>>>>>>>> diff --git a/include/media/videobuf2-core.h > >>>>>>>>> b/include/media/videobuf2-core.h > >>>>>>>>> index 77921cf894ef..080b783d608d 100644 > >>>>>>>>> --- a/include/media/videobuf2-core.h > >>>>>>>>> +++ b/include/media/videobuf2-core.h > >>>>>>>>> @@ -20,7 +20,6 @@ > >>>>>>>>> #include <media/media-request.h> > >>>>>>>>> #include <media/frame_vector.h> > >>>>>>>>> > >>>>>>>>> -#define VB2_MAX_FRAME (32) > >>>>>>>>> #define VB2_MAX_PLANES (8) > >>>>>>>>> > >>>>>>>>> /** > >>>>>>>>> diff --git a/include/media/videobuf2-v4l2.h > >>>>>>>>> b/include/media/videobuf2-v4l2.h > >>>>>>>>> index 5a845887850b..88a7a565170e 100644 > >>>>>>>>> --- a/include/media/videobuf2-v4l2.h > >>>>>>>>> +++ b/include/media/videobuf2-v4l2.h > >>>>>>>>> @@ -15,10 +15,6 @@ > >>>>>>>>> #include <linux/videodev2.h> > >>>>>>>>> #include <media/videobuf2-core.h> > >>>>>>>>> > >>>>>>>>> -#if VB2_MAX_FRAME != VIDEO_MAX_FRAME > >>>>>>>>> -#error VB2_MAX_FRAME != VIDEO_MAX_FRAME > >>>>>>>>> -#endif > >>>>>>>>> - > >>>>>>>>> #if VB2_MAX_PLANES != VIDEO_MAX_PLANES > >>>>>>>>> #error VB2_MAX_PLANES != VIDEO_MAX_PLANES > >>>>>>>>> #endif > >>>>>>>>> -- > >>>>>>>>> 2.39.2 > >>>>>>>>> > >>>>>> -- > >>>>>> Hsia-Jun(Randy) Li > >>>>>> > >>>> -- > >>>> Hsia-Jun(Randy) Li > >>>> > >> > >> -- > >> Hsia-Jun(Randy) Li > > -- > Hsia-Jun(Randy) Li ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v3 04/11] media: videobuf2: Stop define VB2_MAX_FRAME as global 2023-09-07 4:15 ` Tomasz Figa @ 2023-09-07 6:54 ` Hsia-Jun Li 0 siblings, 0 replies; 27+ messages in thread From: Hsia-Jun Li @ 2023-09-07 6:54 UTC (permalink / raw) To: Tomasz Figa Cc: Benjamin Gaignard, linux-media, linux-kernel, linux-arm-kernel, m.szyprowski, linux-mediatek, linux-arm-msm, hverkuil-cisco, ezequiel, p.zabel, linux-rockchip, mchehab, linux-staging, ming.qian, kernel, gregkh, nicolas.dufresne We are little off topic here. On 9/7/23 12:15, Tomasz Figa wrote: > CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe. > > > On Fri, Jul 28, 2023 at 4:25 PM Hsia-Jun Li <Randy.Li@synaptics.com> wrote: >> >> >> >> On 7/28/23 15:03, Tomasz Figa wrote: >>> CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe. >>> >>> >>> On Fri, Jul 28, 2023 at 3:55 PM Hsia-Jun Li <Randy.Li@synaptics.com> wrote: >>>> >>>> >>>> >>>> On 7/28/23 14:46, Tomasz Figa wrote: >>>>> CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe. >>>>> >>>>> >>>>> On Mon, Jul 17, 2023 at 4:44 PM Hsia-Jun Li <Randy.Li@synaptics.com> wrote: >>>>>> >>>>>> >>>>>> On 7/12/23 18:48, Tomasz Figa wrote: >>>>>>> CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe. >>>>>>> >>>>>>> >>>>>>> On Mon, Jul 03, 2023 at 04:35:30PM +0800, Hsia-Jun Li wrote: >>>>>>>> On 7/3/23 16:09, Benjamin Gaignard wrote: >>>>>>>>> CAUTION: Email originated externally, do not click links or open >>>>>>>>> attachments unless you recognize the sender and know the content is >>>>>>>>> safe. >>>>>>>>> >>>>>>>>> >>>>>>>>> Le 30/06/2023 à 11:51, Hsia-Jun Li a écrit : >>>>>>>>>> On 6/22/23 21:13, Benjamin Gaignard wrote: >>>>>>>>>>> CAUTION: Email originated externally, do not click links or open >>>>>>>>>>> attachments unless you recognize the sender and know the content is >>>>>>>>>>> safe. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> After changing bufs arrays to a dynamic allocated array >>>>>>>>>>> VB2_MAX_FRAME doesn't mean anything for videobuf2 core. >>>>>>>>>> I think make it 64 which is the VB2_MAX_FRAME in Android GKI kernel is >>>>>>>>>> more reasonable. >>>>>>>>>> >>>>>>>>>> It would be hard to iterate the whole array, it would go worse with a >>>>>>>>>> filter. Such iterate may need to go twice because you mix >>>>>>>>>> post-processing buffer and decoding buffer(with MV) in the same array. >>>>>>>>> Here I don't want to change drivers behavior so I keep the same value. >>>>>>>>> If it happens that they need more buffers, like for dynamic resolution >>>>>>>>> change >>>>>>>>> feature for Verisilicon VP9 decoder, case by case patches will be needed. >>>>>>>>> >>>>>>>> I just don't like the idea that using a variant length array here. >>>>>>>> >>>>>>> "I don't like" is not an argument. We had a number of arguments for >>>>>>> using a generic helper (originally idr, but later decided to go with >>>>>>> XArray, because the former is now deprecated) that we pointed out in >>>>>>> our review comments for previous revisions. It wasn't really about the >>>>>>> size being variable, but rather avoiding open coding things in vb2 and >>>>>>> duplicating what's already implemented in generic code. >>>>>> >>>>>> I just want to say I don't think we need a variable length array to >>>>>> store the buffer here. >>>>>> >>>>>> And the below is the reason that such a case could be avoided in the >>>>>> first place. >>>>>> >>>>>>> >>>>>>>> And I could explain why you won't need so many buffers for the performance >>>>>>>> of decoding. >>>>>>>> >>>>>>>> VP9 could support 10 reference frames in dpb. >>>>>>>> >>>>>>>> Even for those frequent resolution changing test set, it would only happen >>>>>>>> to two resolutions, >>>>>>>> >>>>>>>> 32 would be enough for 20 buffers of two resolution plus golden frames. It >>>>>>>> also leaves enough slots for re-order latency. >>>>>>>> >>>>>>>> If your case had more two resolutions, likes low->medium->high. >>>>>>>> >>>>>>>> I would suggest just skip the medium resolutions, just allocate the lower >>>>>>>> one first for fast playback then the highest for all the possible >>>>>>>> >>>>>>>> medium cases. Reallocation happens frequently would only cause memory >>>>>>>> fragment, nothing benefits your performance. >>>>>>>> >>>>>>> We have mechanisms in the kernel to deal with memory fragmentation >>>>>>> (migration/compaction) and it would still only matters for the >>>>>>> pathologic cases of hardware that require physically contiguous memory. >>>>>>> Modern hardware with proper DMA capabilities can either scatter-gather >>>>>>> or are equipped with an IOMMU, so the allocation always happens in page >>>>>>> granularity and fragmentation is avoided. >>>>>> >>>>>> Unfortunately, there are more devices that didn't have a IOMMU attached >>>>>> to it, supporting scatter gather is more odd. >>>>>> >>>>>> It would be more likely that IOMMU would be disabled for the performance >>>>>> reason. >>>>> >>>>> These days IOMMU is totally mandatory if you want to think about >>>>> having any level of security in your system. Sure, there could be some >>>>> systems that are completely isolated from any external environment, >>>>> like some offline industry automation machines, but then arguably >>>>> their running conditions would also be quite static and require very >>>>> little memory re-allocation. >>>> Vendor just decided not to included such hardware. >>>> That is why From ION to DMA-heap, people like to allocate from a cavout >>>> out memory. >>>>> >>>>> I also don't buy the performance reason. CPUs have been behind MMUs >>>>> for ages and nobody is running them with paging disabled for >>>>> performance reasons. Similarly, most of the modern consumer systems >>>> Page lookup would increase the delay. Besides a few upstream devices >>>> prove them only use a level 1 page table without TBL. >>> >>> That's just an excuse for a bad hardware design/implementation. As I >>> said, there are good IOMMU implementations out there that don't suffer >>> from performance issues. >>> >> I could do nothing about that. >> Besides, even with TLB, cache missing could happen frequently, >> especially we need to access many (5~16, 10 usually) buffers and more >> 11MBytes each in a hardware processing. >> You can't have a very large TLB. > > Right, but as I wrote in my previous emails, we have the right methods > in the kernel for providing drivers with contiguous memory and those > can be used for those special cases. > I think we were talking about whether the IOMMU should be considered mandatory for the future possible hardware. About the right methods, I don't think we have a protocol for negotiate the allocation hints between drivers or anybody. I sent an email to dri-devel to start(continue) the discussion of that. >>>>> (mobile phones, PCs) run with IOMMUs enabled for pretty much anything >>>>> because of the security reason and they don't seem to be having any >>>> If the page is secure, you can't operate it in a insecure IOMMU or MMU. >>>> The most security way here, we should use a dedicated memory(or a zone >>>> in unified memory). >>> >>> You still need something to enforce that the hardware is not accessing >>> memory that it's not supposed to access. How do you do that without an >>> IOMMU? >>> >> If you know the arm security pipeline and security controller, you could >> found we could reserved a range of memory for a security id(devices in >> secure world may be a different security domain). >> Besides, a MMU or security MPU could mark some pages for the secure >> world access only, it doesn't mean the device need an IOMMU to access >> them. The MPU could filter the access through the AXI id. >>>> I believe there are more users in mobile for DMA-heap than kernel's dma >>>> allocation API. >>> >>> Yes, but that's completely separate from whether there is an IOMMU or >>> not. It's just a different allocation API. >>> >> The memory heap would mean a dedicated memory usually(we don't talk >> about system heap or why there are many vendor heaps). Dedicated memory >> means contiguous memory in the most of cases. > > No, and no. > First no - DMA-buf heap doesn't imply dedicated memory and usually one > wants to completely avoid carving out memory, because it becomes > useless if specific use case is not active. > Second no - there are ways to provide dedicated memory regions to the > DMA mapping API, such as shared or restricted DMA pool [1]. > > [1] https://elixir.bootlin.com/linux/latest/source/Documentation/devicetree/bindings/reserved-memory/shared-dma-pool.yaml > I think the reserved memory exits even before the ION was in kernel. Why Android prefer to use ION and now DMA-heap, I thinik it is a google's problem. Youl can say what reserved memory didn't do is allowing allocate from the userspace. That is how gralloc in Android works. Allocation decided by the users could be a important feature, for example, a video codec could decode either secure or non secure buffer. We could make the driver only allocate the non secure memory while the usersapce could feed the secure memory, that could satisfy both generic application and DRM video playback. > Best regards, > Tomasz > >>>>> performance issues. In fact, it can improve the performance, because >>>>> memory allocation is much easier and without contiguous careouts (as >>>>> we used to have long ago on Android devices) the extra memory can be >>>>> used for buffers and caches to improve system performance. >>>>> >>>>> Best regards, >>>>> Tomasz >>>>> >>>>>> >>>>>>> >>>>>>> Best regards, >>>>>>> Tomasz >>>>>>> >>>>>>>>>>> Remove it from the core definitions but keep it for drivers internal >>>>>>>>>>> needs. >>>>>>>>>>> >>>>>>>>>>> Signed-off-by: Benjamin Gaignard <benjamin.gaignard@collabora.com> >>>>>>>>>>> --- >>>>>>>>>>> drivers/media/common/videobuf2/videobuf2-core.c | 2 ++ >>>>>>>>>>> drivers/media/platform/amphion/vdec.c | 1 + >>>>>>>>>>> .../media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c | 2 ++ >>>>>>>>>>> drivers/media/platform/qcom/venus/hfi.h | 2 ++ >>>>>>>>>>> drivers/media/platform/verisilicon/hantro_hw.h | 2 ++ >>>>>>>>>>> drivers/staging/media/ipu3/ipu3-v4l2.c | 2 ++ >>>>>>>>>>> include/media/videobuf2-core.h | 1 - >>>>>>>>>>> include/media/videobuf2-v4l2.h | 4 ---- >>>>>>>>>>> 8 files changed, 11 insertions(+), 5 deletions(-) >>>>>>>>>>> >>>>>>>>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>>>>> b/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>>>>> index 86e1e926fa45..899783f67580 100644 >>>>>>>>>>> --- a/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>>>>> +++ b/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>>>>> @@ -31,6 +31,8 @@ >>>>>>>>>>> >>>>>>>>>>> #include <trace/events/vb2.h> >>>>>>>>>>> >>>>>>>>>>> +#define VB2_MAX_FRAME 32 >>>>>>>>>>> + >>>>>>>>>>> static int debug; >>>>>>>>>>> module_param(debug, int, 0644); >>>>>>>>>>> >>>>>>>>>>> diff --git a/drivers/media/platform/amphion/vdec.c >>>>>>>>>>> b/drivers/media/platform/amphion/vdec.c >>>>>>>>>>> index 3fa1a74a2e20..b3219f6d17fa 100644 >>>>>>>>>>> --- a/drivers/media/platform/amphion/vdec.c >>>>>>>>>>> +++ b/drivers/media/platform/amphion/vdec.c >>>>>>>>>>> @@ -28,6 +28,7 @@ >>>>>>>>>>> >>>>>>>>>>> #define VDEC_MIN_BUFFER_CAP 8 >>>>>>>>>>> #define VDEC_MIN_BUFFER_OUT 8 >>>>>>>>>>> +#define VB2_MAX_FRAME 32 >>>>>>>>>>> >>>>>>>>>>> struct vdec_fs_info { >>>>>>>>>>> char name[8]; >>>>>>>>>>> diff --git >>>>>>>>>>> a/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >>>>>>>>>>> b/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >>>>>>>>>>> index 6532a69f1fa8..a1e0f24bb91c 100644 >>>>>>>>>>> --- a/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >>>>>>>>>>> +++ b/drivers/media/platform/mediatek/vcodec/vdec/vdec_vp9_req_lat_if.c >>>>>>>>>>> @@ -16,6 +16,8 @@ >>>>>>>>>>> #include "../vdec_drv_if.h" >>>>>>>>>>> #include "../vdec_vpu_if.h" >>>>>>>>>>> >>>>>>>>>>> +#define VB2_MAX_FRAME 32 >>>>>>>>>>> + >>>>>>>>>>> /* reset_frame_context defined in VP9 spec */ >>>>>>>>>>> #define VP9_RESET_FRAME_CONTEXT_NONE0 0 >>>>>>>>>>> #define VP9_RESET_FRAME_CONTEXT_NONE1 1 >>>>>>>>>>> diff --git a/drivers/media/platform/qcom/venus/hfi.h >>>>>>>>>>> b/drivers/media/platform/qcom/venus/hfi.h >>>>>>>>>>> index f25d412d6553..bd5ca5a8b945 100644 >>>>>>>>>>> --- a/drivers/media/platform/qcom/venus/hfi.h >>>>>>>>>>> +++ b/drivers/media/platform/qcom/venus/hfi.h >>>>>>>>>>> @@ -10,6 +10,8 @@ >>>>>>>>>>> >>>>>>>>>>> #include "hfi_helper.h" >>>>>>>>>>> >>>>>>>>>>> +#define VB2_MAX_FRAME 32 >>>>>>>>>>> + >>>>>>>>>>> #define VIDC_SESSION_TYPE_VPE 0 >>>>>>>>>>> #define VIDC_SESSION_TYPE_ENC 1 >>>>>>>>>>> #define VIDC_SESSION_TYPE_DEC 2 >>>>>>>>>>> diff --git a/drivers/media/platform/verisilicon/hantro_hw.h >>>>>>>>>>> b/drivers/media/platform/verisilicon/hantro_hw.h >>>>>>>>>>> index e83f0c523a30..9e8faf7ba6fb 100644 >>>>>>>>>>> --- a/drivers/media/platform/verisilicon/hantro_hw.h >>>>>>>>>>> +++ b/drivers/media/platform/verisilicon/hantro_hw.h >>>>>>>>>>> @@ -15,6 +15,8 @@ >>>>>>>>>>> #include <media/v4l2-vp9.h> >>>>>>>>>>> #include <media/videobuf2-core.h> >>>>>>>>>>> >>>>>>>>>>> +#define VB2_MAX_FRAME 32 >>>>>>>>>>> + >>>>>>>>>>> #define DEC_8190_ALIGN_MASK 0x07U >>>>>>>>>>> >>>>>>>>>>> #define MB_DIM 16 >>>>>>>>>>> diff --git a/drivers/staging/media/ipu3/ipu3-v4l2.c >>>>>>>>>>> b/drivers/staging/media/ipu3/ipu3-v4l2.c >>>>>>>>>>> index e530767e80a5..6627b5c2d4d6 100644 >>>>>>>>>>> --- a/drivers/staging/media/ipu3/ipu3-v4l2.c >>>>>>>>>>> +++ b/drivers/staging/media/ipu3/ipu3-v4l2.c >>>>>>>>>>> @@ -10,6 +10,8 @@ >>>>>>>>>>> #include "ipu3.h" >>>>>>>>>>> #include "ipu3-dmamap.h" >>>>>>>>>>> >>>>>>>>>>> +#define VB2_MAX_FRAME 32 >>>>>>>>>>> + >>>>>>>>>>> /******************** v4l2_subdev_ops ********************/ >>>>>>>>>>> >>>>>>>>>>> #define IPU3_RUNNING_MODE_VIDEO 0 >>>>>>>>>>> diff --git a/include/media/videobuf2-core.h >>>>>>>>>>> b/include/media/videobuf2-core.h >>>>>>>>>>> index 77921cf894ef..080b783d608d 100644 >>>>>>>>>>> --- a/include/media/videobuf2-core.h >>>>>>>>>>> +++ b/include/media/videobuf2-core.h >>>>>>>>>>> @@ -20,7 +20,6 @@ >>>>>>>>>>> #include <media/media-request.h> >>>>>>>>>>> #include <media/frame_vector.h> >>>>>>>>>>> >>>>>>>>>>> -#define VB2_MAX_FRAME (32) >>>>>>>>>>> #define VB2_MAX_PLANES (8) >>>>>>>>>>> >>>>>>>>>>> /** >>>>>>>>>>> diff --git a/include/media/videobuf2-v4l2.h >>>>>>>>>>> b/include/media/videobuf2-v4l2.h >>>>>>>>>>> index 5a845887850b..88a7a565170e 100644 >>>>>>>>>>> --- a/include/media/videobuf2-v4l2.h >>>>>>>>>>> +++ b/include/media/videobuf2-v4l2.h >>>>>>>>>>> @@ -15,10 +15,6 @@ >>>>>>>>>>> #include <linux/videodev2.h> >>>>>>>>>>> #include <media/videobuf2-core.h> >>>>>>>>>>> >>>>>>>>>>> -#if VB2_MAX_FRAME != VIDEO_MAX_FRAME >>>>>>>>>>> -#error VB2_MAX_FRAME != VIDEO_MAX_FRAME >>>>>>>>>>> -#endif >>>>>>>>>>> - >>>>>>>>>>> #if VB2_MAX_PLANES != VIDEO_MAX_PLANES >>>>>>>>>>> #error VB2_MAX_PLANES != VIDEO_MAX_PLANES >>>>>>>>>>> #endif >>>>>>>>>>> -- >>>>>>>>>>> 2.39.2 >>>>>>>>>>> >>>>>>>> -- >>>>>>>> Hsia-Jun(Randy) Li >>>>>>>> >>>>>> -- >>>>>> Hsia-Jun(Randy) Li >>>>>> >>>> >>>> -- >>>> Hsia-Jun(Randy) Li >> >> -- >> Hsia-Jun(Randy) Li -- Hsia-Jun(Randy) Li ^ permalink raw reply [flat|nested] 27+ messages in thread
[parent not found: <20230622131349.144160-11-benjamin.gaignard@collabora.com>]
[parent not found: <80a03c29-6f3d-43f1-755c-10f3ae2c2756@synaptics.com>]
[parent not found: <42a45bad-09c7-ffb6-49cb-29ec0826599d@collabora.com>]
[parent not found: <3b22fda9-4a0e-2c03-2f12-19e5e3239235@synaptics.com>]
* Re: [PATCH v3 10/11] media: v4l2: Add DELETE_BUF ioctl [not found] ` <3b22fda9-4a0e-2c03-2f12-19e5e3239235@synaptics.com> @ 2023-06-30 9:43 ` Hsia-Jun Li 2023-07-03 8:12 ` Benjamin Gaignard 2023-07-13 9:09 ` Tomasz Figa 0 siblings, 2 replies; 27+ messages in thread From: Hsia-Jun Li @ 2023-06-30 9:43 UTC (permalink / raw) To: Benjamin Gaignard Cc: linux-media, linux-kernel, p.zabel, ming.qian, hverkuil-cisco, gregkh, ezequiel, linux-arm-kernel, linux-mediatek, linux-arm-msm, linux-rockchip, nicolas.dufresne, linux-staging, kernel, mchehab, tfiga, m.szyprowski, ayaka On 6/27/23 16:47, Hsia-Jun Li wrote: > CAUTION: Email originated externally, do not click links or open > attachments unless you recognize the sender and know the content is safe. > > > On 6/27/23 16:43, Benjamin Gaignard wrote: >> CAUTION: Email originated externally, do not click links or open >> attachments unless you recognize the sender and know the content is >> safe. >> >> >> Le 27/06/2023 à 09:30, Hsia-Jun Li a écrit : >>> >>> On 6/22/23 21:13, Benjamin Gaignard wrote: >>>> CAUTION: Email originated externally, do not click links or open >>>> attachments unless you recognize the sender and know the content is >>>> safe. >>>> >>>> >>>> VIDIOC_DELETE_BUF ioctl allows to delete a buffer from a queue. >>>> >>>> Signed-off-by: Benjamin Gaignard <benjamin.gaignard@collabora.com> >>>> --- >>>> .../userspace-api/media/v4l/user-func.rst | 1 + >>>> .../media/v4l/vidioc-delete-buf.rst | 51 >>>> +++++++++++++++++++ >>>> .../media/common/videobuf2/videobuf2-core.c | 33 ++++++++++++ >>>> .../media/common/videobuf2/videobuf2-v4l2.c | 6 +++ >>>> drivers/media/v4l2-core/v4l2-dev.c | 1 + >>>> drivers/media/v4l2-core/v4l2-ioctl.c | 10 ++++ >>>> include/media/v4l2-ioctl.h | 4 ++ >>>> include/media/videobuf2-core.h | 9 ++++ >>>> include/media/videobuf2-v4l2.h | 11 ++++ >>>> include/uapi/linux/videodev2.h | 2 + >>>> 10 files changed, 128 insertions(+) >>>> create mode 100644 >>>> Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>> >>>> diff --git a/Documentation/userspace-api/media/v4l/user-func.rst >>>> b/Documentation/userspace-api/media/v4l/user-func.rst >>>> index 15ff0bf7bbe6..8c74016e12fd 100644 >>>> --- a/Documentation/userspace-api/media/v4l/user-func.rst >>>> +++ b/Documentation/userspace-api/media/v4l/user-func.rst >>>> @@ -17,6 +17,7 @@ Function Reference >>>> vidioc-dbg-g-chip-info >>>> vidioc-dbg-g-register >>>> vidioc-decoder-cmd >>>> + vidioc-delete-buf >>>> vidioc-dqevent >>>> vidioc-dv-timings-cap >>>> vidioc-encoder-cmd >>>> diff --git >>>> a/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>> b/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>> new file mode 100644 >>>> index 000000000000..0e7ce58f91bc >>>> --- /dev/null >>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>> @@ -0,0 +1,51 @@ >>>> +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later >>>> +.. c:namespace:: V4L >>>> + >>>> +.. _VIDIOC_DELETE_BUF: >>>> + >>>> +************************ >>>> +ioctl VIDIOC_DELETE_BUF >>>> +************************ >>>> + >>>> +Name >>>> +==== >>>> + >>>> +VIDIOC_DELETE_BUF - Delete a buffer from a queue >>>> + >>>> +Synopsis >>>> +======== >>>> + >>>> +.. c:macro:: VIDIOC_DELETE_BUF >>>> + >>>> +``int ioctl(int fd, VIDIOC_DELETE_BUF, struct v4l2_buffer *argp)`` >>>> + >>>> +Arguments >>>> +========= >>>> + >>>> +``fd`` >>>> + File descriptor returned by :c:func:`open()`. >>>> + >>>> +``argp`` >>>> + Pointer to struct :c:type:`v4l2_buffer`. >>>> + >>>> +Description >>>> +=========== >>>> + >>>> +Applications can optionally call the :ref:`VIDIOC_DELETE_BUF` >>>> ioctl to >>>> +delete a buffer from a queue. >>>> + >>>> +The struct :c:type:`v4l2_buffer` structure is specified in >>>> +:ref:`buffer`. >>>> + >>>> +Return Value >>>> +============ >>>> + >>>> +On success 0 is returned, on error -1 and the ``errno`` variable is >>>> set >>>> +appropriately. The generic error codes are described at the >>>> +:ref:`Generic Error Codes <gen-errors>` chapter. >>>> + >>>> +EBUSY >>>> + File I/O is in progress. >>>> + >>>> +EINVAL >>>> + The buffer ``index`` doesn't exist in the queue. >>>> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c >>>> b/drivers/media/common/videobuf2/videobuf2-core.c >>>> index 899783f67580..aa546c972c3d 100644 >>>> --- a/drivers/media/common/videobuf2/videobuf2-core.c >>>> +++ b/drivers/media/common/videobuf2/videobuf2-core.c >>>> @@ -1637,6 +1637,39 @@ int vb2_core_prepare_buf(struct vb2_queue *q, >>>> unsigned int index, void *pb) >>>> } >>>> EXPORT_SYMBOL_GPL(vb2_core_prepare_buf); >>>> >>>> +int vb2_core_delete_buf(struct vb2_queue *q, unsigned int index) >>>> +{ >>>> + struct vb2_buffer *vb; >>>> + >>>> + vb = vb2_get_buffer(q, index); >>>> + if (!vb) { >>>> + dprintk(q, 1, "invalid buffer index %d\n", index); >>>> + return -EINVAL; >>>> + } >>>> + >>>> + if (vb->state != VB2_BUF_STATE_DEQUEUED) { >>>> + dprintk(q, 1, "can't delete non dequeued buffer index >>>> %d\n", index); >>>> + return -EINVAL; >>>> + } >>>> + >>> I know the driver could implement its own >>> v4l2_ioctl_ops->vidioc_delete_buf() that check whether a buffer is >>> used by the hardware as a future reference frame. >>> But I think we need a flag to let the user know which buffer is still >>> used by the hardware. >>> Alternative ref case is safe, we only know it's existing when it is >>> dequeued in current V4L2 buffer mechanism. >>> While the Golden reference frame, such long term reference frame could >>> last much longer. >> >> It is up to userland stack to know frames life time, it got the >> information for that. > > That is true for the stateless codec driver. > > While application for stateful decoder could never do that. It also > breaks what the document said: > > "The backing memory of |CAPTURE| buffers that are used as reference > frames by the stream may be read by the hardware even after they are > dequeued. Consequently, the client should avoid writing into this memory > while the |CAPTURE| queue is streaming. Failure to observe this may > result in corruption of decoded frames." > >> >>>> + if (vb->planes[0].mem_priv) >>>> + call_void_vb_qop(vb, buf_cleanup, vb); >>>> + >>>> + /* Free MMAP buffers or release USERPTR buffers */ >>>> + if (q->memory == VB2_MEMORY_MMAP) >>>> + __vb2_buf_mem_free(vb); Here is another problem for the existing application, the mmap() from the mmap offset or exportbuffer fd would not create a reference to buffer in this step(while the exportbuffer would create one itself). When you delete a buffer, you may not release it from its virtual memory space, leaving a corrupted virtual memory space. Also this behavior is right, because mmap(2) says: "After the mmap() call has returned, the file descriptor, fd, can be closed immediately without invalidating the map‐ping." >>>> + else if (q->memory == VB2_MEMORY_DMABUF) >>>> + __vb2_buf_dmabuf_put(vb); >>>> + else >>>> + __vb2_buf_userptr_put(vb); >>>> + >>>> + vb2_queue_remove_buffer(q, vb); >>>> + kfree(vb); >>>> + >>>> + dprintk(q, 2, "buffer %d deleted\n", index); >>>> + return 0; >>>> +} >>>> + >>>> /* >>>> * vb2_start_streaming() - Attempt to start streaming. >>>> * @q: videobuf2 queue >>>> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>> b/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>> index 724135d41f7f..cea666c17b41 100644 >>>> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>> @@ -751,6 +751,12 @@ int vb2_prepare_buf(struct vb2_queue *q, struct >>>> media_device *mdev, >>>> } >>>> EXPORT_SYMBOL_GPL(vb2_prepare_buf); >>>> >>>> +int vb2_delete_buf(struct vb2_queue *q, struct v4l2_buffer *b) >>>> +{ >>>> + return vb2_core_delete_buf(q, b->index); >>>> +} >>>> +EXPORT_SYMBOL_GPL(vb2_delete_buf); >>>> + >>>> int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers >>>> *create) >>>> { >>>> unsigned requested_planes = 1; >>>> diff --git a/drivers/media/v4l2-core/v4l2-dev.c >>>> b/drivers/media/v4l2-core/v4l2-dev.c >>>> index f81279492682..80ace2e1e932 100644 >>>> --- a/drivers/media/v4l2-core/v4l2-dev.c >>>> +++ b/drivers/media/v4l2-core/v4l2-dev.c >>>> @@ -720,6 +720,7 @@ static void determine_valid_ioctls(struct >>>> video_device *vdev) >>>> SET_VALID_IOCTL(ops, VIDIOC_PREPARE_BUF, >>>> vidioc_prepare_buf); >>>> SET_VALID_IOCTL(ops, VIDIOC_STREAMON, >>>> vidioc_streamon); >>>> SET_VALID_IOCTL(ops, VIDIOC_STREAMOFF, >>>> vidioc_streamoff); >>>> + SET_VALID_IOCTL(ops, VIDIOC_DELETE_BUF, >>>> vidioc_delete_buf); >>>> } >>>> >>>> if (is_vid || is_vbi || is_meta) { >>>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c >>>> b/drivers/media/v4l2-core/v4l2-ioctl.c >>>> index a858acea6547..1c737279d3ef 100644 >>>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c >>>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c >>>> @@ -2156,6 +2156,15 @@ static int v4l_prepare_buf(const struct >>>> v4l2_ioctl_ops *ops, >>>> return ret ? ret : ops->vidioc_prepare_buf(file, fh, b); >>>> } >>>> >>>> +static int v4l_delete_buf(const struct v4l2_ioctl_ops *ops, >>>> + struct file *file, void *fh, void *arg) >>>> +{ >>>> + struct v4l2_buffer *b = arg; >>>> + int ret = check_fmt(file, b->type); >>>> + >>>> + return ret ? ret : ops->vidioc_delete_buf(file, fh, b); >>>> +} >>>> + >>>> static int v4l_g_parm(const struct v4l2_ioctl_ops *ops, >>>> struct file *file, void *fh, void >>>> *arg) >>>> { >>>> @@ -2905,6 +2914,7 @@ static const struct v4l2_ioctl_info >>>> v4l2_ioctls[] = { >>>> IOCTL_INFO(VIDIOC_ENUM_FREQ_BANDS, v4l_enum_freq_bands, >>>> v4l_print_freq_band, 0), >>>> IOCTL_INFO(VIDIOC_DBG_G_CHIP_INFO, v4l_dbg_g_chip_info, >>>> v4l_print_dbg_chip_info, INFO_FL_CLEAR(v4l2_dbg_chip_info, match)), >>>> IOCTL_INFO(VIDIOC_QUERY_EXT_CTRL, v4l_query_ext_ctrl, >>>> v4l_print_query_ext_ctrl, INFO_FL_CTRL | >>>> INFO_FL_CLEAR(v4l2_query_ext_ctrl, id)), >>>> + IOCTL_INFO(VIDIOC_DELETE_BUF, v4l_delete_buf, >>>> v4l_print_buffer, INFO_FL_QUEUE), >>>> }; >>>> #define V4L2_IOCTLS ARRAY_SIZE(v4l2_ioctls) >>>> >>>> diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h >>>> index edb733f21604..2f232ed884c7 100644 >>>> --- a/include/media/v4l2-ioctl.h >>>> +++ b/include/media/v4l2-ioctl.h >>>> @@ -163,6 +163,8 @@ struct v4l2_fh; >>>> * :ref:`VIDIOC_CREATE_BUFS <vidioc_create_bufs>` ioctl >>>> * @vidioc_prepare_buf: pointer to the function that implements >>>> * :ref:`VIDIOC_PREPARE_BUF <vidioc_prepare_buf>` ioctl >>>> + * @vidioc_delete_buf: pointer to the function that implements >>>> + * :ref:`VIDIOC_DELETE_BUF <vidioc_delete_buf>` ioctl >>>> * @vidioc_overlay: pointer to the function that implements >>>> * :ref:`VIDIOC_OVERLAY <vidioc_overlay>` ioctl >>>> * @vidioc_g_fbuf: pointer to the function that implements >>>> @@ -422,6 +424,8 @@ struct v4l2_ioctl_ops { >>>> struct v4l2_create_buffers *b); >>>> int (*vidioc_prepare_buf)(struct file *file, void *fh, >>>> struct v4l2_buffer *b); >>>> + int (*vidioc_delete_buf)(struct file *file, void *fh, >>>> + struct v4l2_buffer *b); >>>> >>>> int (*vidioc_overlay)(struct file *file, void *fh, unsigned >>>> int i); >>>> int (*vidioc_g_fbuf)(struct file *file, void *fh, >>>> diff --git a/include/media/videobuf2-core.h >>>> b/include/media/videobuf2-core.h >>>> index 080b783d608d..0f9e68f76b77 100644 >>>> --- a/include/media/videobuf2-core.h >>>> +++ b/include/media/videobuf2-core.h >>>> @@ -840,6 +840,15 @@ int vb2_core_create_bufs(struct vb2_queue *q, >>>> enum vb2_memory memory, >>>> */ >>>> int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, >>>> void *pb); >>>> >>>> +/** >>>> + * vb2_core_delete_buf() - >>>> + * @q: pointer to &struct vb2_queue with videobuf2 queue. >>>> + * @index: id number of the buffer. >>>> + * >>>> + * Return: returns zero on success; an error code otherwise. >>>> + */ >>>> +int vb2_core_delete_buf(struct vb2_queue *q, unsigned int index); >>>> + >>>> /** >>>> * vb2_core_qbuf() - Queue a buffer from userspace >>>> * >>>> diff --git a/include/media/videobuf2-v4l2.h >>>> b/include/media/videobuf2-v4l2.h >>>> index 88a7a565170e..3beeb4c735f0 100644 >>>> --- a/include/media/videobuf2-v4l2.h >>>> +++ b/include/media/videobuf2-v4l2.h >>>> @@ -114,6 +114,17 @@ int vb2_create_bufs(struct vb2_queue *q, struct >>>> v4l2_create_buffers *create); >>>> */ >>>> int vb2_prepare_buf(struct vb2_queue *q, struct media_device *mdev, >>>> struct v4l2_buffer *b); >>>> +/** >>>> + * vb2_delete_buf() - Delete the buffer from the queue >>>> + * >>>> + * @q: pointer to &struct vb2_queue with videobuf2 queue. >>>> + * @b: buffer structure passed from userspace to >>>> + * &v4l2_ioctl_ops->vidioc_delete_buf handler in driver >>>> + * >>>> + * The return values from this function are intended to be directly >>>> returned >>>> + * from &v4l2_ioctl_ops->vidioc_delete_buf handler in driver. >>>> + */ >>>> +int vb2_delete_buf(struct vb2_queue *q, struct v4l2_buffer *b); >>>> >>>> /** >>>> * vb2_qbuf() - Queue a buffer from userspace >>>> diff --git a/include/uapi/linux/videodev2.h >>>> b/include/uapi/linux/videodev2.h >>>> index aee75eb9e686..31bba1915642 100644 >>>> --- a/include/uapi/linux/videodev2.h >>>> +++ b/include/uapi/linux/videodev2.h >>>> @@ -2702,6 +2702,8 @@ struct v4l2_create_buffers { >>>> #define VIDIOC_DBG_G_CHIP_INFO _IOWR('V', 102, struct >>>> v4l2_dbg_chip_info) >>>> >>>> #define VIDIOC_QUERY_EXT_CTRL _IOWR('V', 103, struct >>>> v4l2_query_ext_ctrl) >>>> +#define VIDIOC_DELETE_BUF _IOWR('V', 104, struct v4l2_buffer) >>>> + >>>> >>>> /* Reminder: when adding new ioctls please add support for them to >>>> drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */ >>>> -- >>>> 2.39.2 >>>> > -- > Hsia-Jun(Randy) Li > -- Hsia-Jun(Randy) Li ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v3 10/11] media: v4l2: Add DELETE_BUF ioctl 2023-06-30 9:43 ` [PATCH v3 10/11] media: v4l2: Add DELETE_BUF ioctl Hsia-Jun Li @ 2023-07-03 8:12 ` Benjamin Gaignard 2023-07-03 8:19 ` Hsia-Jun Li 2023-07-13 9:09 ` Tomasz Figa 1 sibling, 1 reply; 27+ messages in thread From: Benjamin Gaignard @ 2023-07-03 8:12 UTC (permalink / raw) To: Hsia-Jun Li Cc: linux-media, linux-kernel, p.zabel, ming.qian, hverkuil-cisco, gregkh, ezequiel, linux-arm-kernel, linux-mediatek, linux-arm-msm, linux-rockchip, nicolas.dufresne, linux-staging, kernel, mchehab, tfiga, m.szyprowski, ayaka Le 30/06/2023 à 11:43, Hsia-Jun Li a écrit : > > On 6/27/23 16:47, Hsia-Jun Li wrote: >> CAUTION: Email originated externally, do not click links or open >> attachments unless you recognize the sender and know the content is >> safe. >> >> >> On 6/27/23 16:43, Benjamin Gaignard wrote: >>> CAUTION: Email originated externally, do not click links or open >>> attachments unless you recognize the sender and know the content is >>> safe. >>> >>> >>> Le 27/06/2023 à 09:30, Hsia-Jun Li a écrit : >>>> >>>> On 6/22/23 21:13, Benjamin Gaignard wrote: >>>>> CAUTION: Email originated externally, do not click links or open >>>>> attachments unless you recognize the sender and know the content is >>>>> safe. >>>>> >>>>> >>>>> VIDIOC_DELETE_BUF ioctl allows to delete a buffer from a queue. >>>>> >>>>> Signed-off-by: Benjamin Gaignard <benjamin.gaignard@collabora.com> >>>>> --- >>>>> .../userspace-api/media/v4l/user-func.rst | 1 + >>>>> .../media/v4l/vidioc-delete-buf.rst | 51 >>>>> +++++++++++++++++++ >>>>> .../media/common/videobuf2/videobuf2-core.c | 33 ++++++++++++ >>>>> .../media/common/videobuf2/videobuf2-v4l2.c | 6 +++ >>>>> drivers/media/v4l2-core/v4l2-dev.c | 1 + >>>>> drivers/media/v4l2-core/v4l2-ioctl.c | 10 ++++ >>>>> include/media/v4l2-ioctl.h | 4 ++ >>>>> include/media/videobuf2-core.h | 9 ++++ >>>>> include/media/videobuf2-v4l2.h | 11 ++++ >>>>> include/uapi/linux/videodev2.h | 2 + >>>>> 10 files changed, 128 insertions(+) >>>>> create mode 100644 >>>>> Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>> >>>>> diff --git a/Documentation/userspace-api/media/v4l/user-func.rst >>>>> b/Documentation/userspace-api/media/v4l/user-func.rst >>>>> index 15ff0bf7bbe6..8c74016e12fd 100644 >>>>> --- a/Documentation/userspace-api/media/v4l/user-func.rst >>>>> +++ b/Documentation/userspace-api/media/v4l/user-func.rst >>>>> @@ -17,6 +17,7 @@ Function Reference >>>>> vidioc-dbg-g-chip-info >>>>> vidioc-dbg-g-register >>>>> vidioc-decoder-cmd >>>>> + vidioc-delete-buf >>>>> vidioc-dqevent >>>>> vidioc-dv-timings-cap >>>>> vidioc-encoder-cmd >>>>> diff --git >>>>> a/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>> b/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>> new file mode 100644 >>>>> index 000000000000..0e7ce58f91bc >>>>> --- /dev/null >>>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>> @@ -0,0 +1,51 @@ >>>>> +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later >>>>> +.. c:namespace:: V4L >>>>> + >>>>> +.. _VIDIOC_DELETE_BUF: >>>>> + >>>>> +************************ >>>>> +ioctl VIDIOC_DELETE_BUF >>>>> +************************ >>>>> + >>>>> +Name >>>>> +==== >>>>> + >>>>> +VIDIOC_DELETE_BUF - Delete a buffer from a queue >>>>> + >>>>> +Synopsis >>>>> +======== >>>>> + >>>>> +.. c:macro:: VIDIOC_DELETE_BUF >>>>> + >>>>> +``int ioctl(int fd, VIDIOC_DELETE_BUF, struct v4l2_buffer *argp)`` >>>>> + >>>>> +Arguments >>>>> +========= >>>>> + >>>>> +``fd`` >>>>> + File descriptor returned by :c:func:`open()`. >>>>> + >>>>> +``argp`` >>>>> + Pointer to struct :c:type:`v4l2_buffer`. >>>>> + >>>>> +Description >>>>> +=========== >>>>> + >>>>> +Applications can optionally call the :ref:`VIDIOC_DELETE_BUF` >>>>> ioctl to >>>>> +delete a buffer from a queue. >>>>> + >>>>> +The struct :c:type:`v4l2_buffer` structure is specified in >>>>> +:ref:`buffer`. >>>>> + >>>>> +Return Value >>>>> +============ >>>>> + >>>>> +On success 0 is returned, on error -1 and the ``errno`` variable is >>>>> set >>>>> +appropriately. The generic error codes are described at the >>>>> +:ref:`Generic Error Codes <gen-errors>` chapter. >>>>> + >>>>> +EBUSY >>>>> + File I/O is in progress. >>>>> + >>>>> +EINVAL >>>>> + The buffer ``index`` doesn't exist in the queue. >>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c >>>>> b/drivers/media/common/videobuf2/videobuf2-core.c >>>>> index 899783f67580..aa546c972c3d 100644 >>>>> --- a/drivers/media/common/videobuf2/videobuf2-core.c >>>>> +++ b/drivers/media/common/videobuf2/videobuf2-core.c >>>>> @@ -1637,6 +1637,39 @@ int vb2_core_prepare_buf(struct vb2_queue *q, >>>>> unsigned int index, void *pb) >>>>> } >>>>> EXPORT_SYMBOL_GPL(vb2_core_prepare_buf); >>>>> >>>>> +int vb2_core_delete_buf(struct vb2_queue *q, unsigned int index) >>>>> +{ >>>>> + struct vb2_buffer *vb; >>>>> + >>>>> + vb = vb2_get_buffer(q, index); >>>>> + if (!vb) { >>>>> + dprintk(q, 1, "invalid buffer index %d\n", index); >>>>> + return -EINVAL; >>>>> + } >>>>> + >>>>> + if (vb->state != VB2_BUF_STATE_DEQUEUED) { >>>>> + dprintk(q, 1, "can't delete non dequeued buffer index >>>>> %d\n", index); >>>>> + return -EINVAL; >>>>> + } >>>>> + >>>> I know the driver could implement its own >>>> v4l2_ioctl_ops->vidioc_delete_buf() that check whether a buffer is >>>> used by the hardware as a future reference frame. >>>> But I think we need a flag to let the user know which buffer is still >>>> used by the hardware. >>>> Alternative ref case is safe, we only know it's existing when it is >>>> dequeued in current V4L2 buffer mechanism. >>>> While the Golden reference frame, such long term reference frame could >>>> last much longer. >>> >>> It is up to userland stack to know frames life time, it got the >>> information for that. >> >> That is true for the stateless codec driver. >> >> While application for stateful decoder could never do that. It also >> breaks what the document said: >> >> "The backing memory of |CAPTURE| buffers that are used as reference >> frames by the stream may be read by the hardware even after they are >> dequeued. Consequently, the client should avoid writing into this memory >> while the |CAPTURE| queue is streaming. Failure to observe this may >> result in corruption of decoded frames." >> >>> >>>>> + if (vb->planes[0].mem_priv) >>>>> + call_void_vb_qop(vb, buf_cleanup, vb); >>>>> + >>>>> + /* Free MMAP buffers or release USERPTR buffers */ >>>>> + if (q->memory == VB2_MEMORY_MMAP) >>>>> + __vb2_buf_mem_free(vb); > > Here is another problem for the existing application, the mmap() from > the mmap offset or exportbuffer fd would not create a reference to > buffer in this step(while the exportbuffer would create one itself). > > When you delete a buffer, you may not release it from its virtual > memory space, leaving a corrupted virtual memory space. Also this > behavior is right, because mmap(2) says: > > "After the mmap() call has returned, the file descriptor, fd, can > be closed immediately without invalidating the map‐ping." Existing applications do not call DELETE_BUF ioctl and when call it they will be aware that the buffer is removed. I have done it in GStreamer: https://gitlab.freedesktop.org/benjamin.gaignard1/gstreamer/-/commit/fca0fbc934f4440693ce0ff6c8dc8a2e5f5f17d9 Regards, Benjamin > >>>>> + else if (q->memory == VB2_MEMORY_DMABUF) >>>>> + __vb2_buf_dmabuf_put(vb); >>>>> + else >>>>> + __vb2_buf_userptr_put(vb); >>>>> + >>>>> + vb2_queue_remove_buffer(q, vb); >>>>> + kfree(vb); >>>>> + >>>>> + dprintk(q, 2, "buffer %d deleted\n", index); >>>>> + return 0; >>>>> +} >>>>> + >>>>> /* >>>>> * vb2_start_streaming() - Attempt to start streaming. >>>>> * @q: videobuf2 queue >>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>> b/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>> index 724135d41f7f..cea666c17b41 100644 >>>>> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>> @@ -751,6 +751,12 @@ int vb2_prepare_buf(struct vb2_queue *q, struct >>>>> media_device *mdev, >>>>> } >>>>> EXPORT_SYMBOL_GPL(vb2_prepare_buf); >>>>> >>>>> +int vb2_delete_buf(struct vb2_queue *q, struct v4l2_buffer *b) >>>>> +{ >>>>> + return vb2_core_delete_buf(q, b->index); >>>>> +} >>>>> +EXPORT_SYMBOL_GPL(vb2_delete_buf); >>>>> + >>>>> int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers >>>>> *create) >>>>> { >>>>> unsigned requested_planes = 1; >>>>> diff --git a/drivers/media/v4l2-core/v4l2-dev.c >>>>> b/drivers/media/v4l2-core/v4l2-dev.c >>>>> index f81279492682..80ace2e1e932 100644 >>>>> --- a/drivers/media/v4l2-core/v4l2-dev.c >>>>> +++ b/drivers/media/v4l2-core/v4l2-dev.c >>>>> @@ -720,6 +720,7 @@ static void determine_valid_ioctls(struct >>>>> video_device *vdev) >>>>> SET_VALID_IOCTL(ops, VIDIOC_PREPARE_BUF, >>>>> vidioc_prepare_buf); >>>>> SET_VALID_IOCTL(ops, VIDIOC_STREAMON, >>>>> vidioc_streamon); >>>>> SET_VALID_IOCTL(ops, VIDIOC_STREAMOFF, >>>>> vidioc_streamoff); >>>>> + SET_VALID_IOCTL(ops, VIDIOC_DELETE_BUF, >>>>> vidioc_delete_buf); >>>>> } >>>>> >>>>> if (is_vid || is_vbi || is_meta) { >>>>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c >>>>> b/drivers/media/v4l2-core/v4l2-ioctl.c >>>>> index a858acea6547..1c737279d3ef 100644 >>>>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c >>>>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c >>>>> @@ -2156,6 +2156,15 @@ static int v4l_prepare_buf(const struct >>>>> v4l2_ioctl_ops *ops, >>>>> return ret ? ret : ops->vidioc_prepare_buf(file, fh, b); >>>>> } >>>>> >>>>> +static int v4l_delete_buf(const struct v4l2_ioctl_ops *ops, >>>>> + struct file *file, void *fh, void *arg) >>>>> +{ >>>>> + struct v4l2_buffer *b = arg; >>>>> + int ret = check_fmt(file, b->type); >>>>> + >>>>> + return ret ? ret : ops->vidioc_delete_buf(file, fh, b); >>>>> +} >>>>> + >>>>> static int v4l_g_parm(const struct v4l2_ioctl_ops *ops, >>>>> struct file *file, void *fh, void >>>>> *arg) >>>>> { >>>>> @@ -2905,6 +2914,7 @@ static const struct v4l2_ioctl_info >>>>> v4l2_ioctls[] = { >>>>> IOCTL_INFO(VIDIOC_ENUM_FREQ_BANDS, v4l_enum_freq_bands, >>>>> v4l_print_freq_band, 0), >>>>> IOCTL_INFO(VIDIOC_DBG_G_CHIP_INFO, v4l_dbg_g_chip_info, >>>>> v4l_print_dbg_chip_info, INFO_FL_CLEAR(v4l2_dbg_chip_info, match)), >>>>> IOCTL_INFO(VIDIOC_QUERY_EXT_CTRL, v4l_query_ext_ctrl, >>>>> v4l_print_query_ext_ctrl, INFO_FL_CTRL | >>>>> INFO_FL_CLEAR(v4l2_query_ext_ctrl, id)), >>>>> + IOCTL_INFO(VIDIOC_DELETE_BUF, v4l_delete_buf, >>>>> v4l_print_buffer, INFO_FL_QUEUE), >>>>> }; >>>>> #define V4L2_IOCTLS ARRAY_SIZE(v4l2_ioctls) >>>>> >>>>> diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h >>>>> index edb733f21604..2f232ed884c7 100644 >>>>> --- a/include/media/v4l2-ioctl.h >>>>> +++ b/include/media/v4l2-ioctl.h >>>>> @@ -163,6 +163,8 @@ struct v4l2_fh; >>>>> * :ref:`VIDIOC_CREATE_BUFS <vidioc_create_bufs>` ioctl >>>>> * @vidioc_prepare_buf: pointer to the function that implements >>>>> * :ref:`VIDIOC_PREPARE_BUF <vidioc_prepare_buf>` ioctl >>>>> + * @vidioc_delete_buf: pointer to the function that implements >>>>> + * :ref:`VIDIOC_DELETE_BUF <vidioc_delete_buf>` ioctl >>>>> * @vidioc_overlay: pointer to the function that implements >>>>> * :ref:`VIDIOC_OVERLAY <vidioc_overlay>` ioctl >>>>> * @vidioc_g_fbuf: pointer to the function that implements >>>>> @@ -422,6 +424,8 @@ struct v4l2_ioctl_ops { >>>>> struct v4l2_create_buffers *b); >>>>> int (*vidioc_prepare_buf)(struct file *file, void *fh, >>>>> struct v4l2_buffer *b); >>>>> + int (*vidioc_delete_buf)(struct file *file, void *fh, >>>>> + struct v4l2_buffer *b); >>>>> >>>>> int (*vidioc_overlay)(struct file *file, void *fh, unsigned >>>>> int i); >>>>> int (*vidioc_g_fbuf)(struct file *file, void *fh, >>>>> diff --git a/include/media/videobuf2-core.h >>>>> b/include/media/videobuf2-core.h >>>>> index 080b783d608d..0f9e68f76b77 100644 >>>>> --- a/include/media/videobuf2-core.h >>>>> +++ b/include/media/videobuf2-core.h >>>>> @@ -840,6 +840,15 @@ int vb2_core_create_bufs(struct vb2_queue *q, >>>>> enum vb2_memory memory, >>>>> */ >>>>> int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, >>>>> void *pb); >>>>> >>>>> +/** >>>>> + * vb2_core_delete_buf() - >>>>> + * @q: pointer to &struct vb2_queue with videobuf2 queue. >>>>> + * @index: id number of the buffer. >>>>> + * >>>>> + * Return: returns zero on success; an error code otherwise. >>>>> + */ >>>>> +int vb2_core_delete_buf(struct vb2_queue *q, unsigned int index); >>>>> + >>>>> /** >>>>> * vb2_core_qbuf() - Queue a buffer from userspace >>>>> * >>>>> diff --git a/include/media/videobuf2-v4l2.h >>>>> b/include/media/videobuf2-v4l2.h >>>>> index 88a7a565170e..3beeb4c735f0 100644 >>>>> --- a/include/media/videobuf2-v4l2.h >>>>> +++ b/include/media/videobuf2-v4l2.h >>>>> @@ -114,6 +114,17 @@ int vb2_create_bufs(struct vb2_queue *q, struct >>>>> v4l2_create_buffers *create); >>>>> */ >>>>> int vb2_prepare_buf(struct vb2_queue *q, struct media_device *mdev, >>>>> struct v4l2_buffer *b); >>>>> +/** >>>>> + * vb2_delete_buf() - Delete the buffer from the queue >>>>> + * >>>>> + * @q: pointer to &struct vb2_queue with videobuf2 queue. >>>>> + * @b: buffer structure passed from userspace to >>>>> + * &v4l2_ioctl_ops->vidioc_delete_buf handler in driver >>>>> + * >>>>> + * The return values from this function are intended to be directly >>>>> returned >>>>> + * from &v4l2_ioctl_ops->vidioc_delete_buf handler in driver. >>>>> + */ >>>>> +int vb2_delete_buf(struct vb2_queue *q, struct v4l2_buffer *b); >>>>> >>>>> /** >>>>> * vb2_qbuf() - Queue a buffer from userspace >>>>> diff --git a/include/uapi/linux/videodev2.h >>>>> b/include/uapi/linux/videodev2.h >>>>> index aee75eb9e686..31bba1915642 100644 >>>>> --- a/include/uapi/linux/videodev2.h >>>>> +++ b/include/uapi/linux/videodev2.h >>>>> @@ -2702,6 +2702,8 @@ struct v4l2_create_buffers { >>>>> #define VIDIOC_DBG_G_CHIP_INFO _IOWR('V', 102, struct >>>>> v4l2_dbg_chip_info) >>>>> >>>>> #define VIDIOC_QUERY_EXT_CTRL _IOWR('V', 103, struct >>>>> v4l2_query_ext_ctrl) >>>>> +#define VIDIOC_DELETE_BUF _IOWR('V', 104, struct v4l2_buffer) >>>>> + >>>>> >>>>> /* Reminder: when adding new ioctls please add support for them to >>>>> drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */ >>>>> -- >>>>> 2.39.2 >>>>> >> -- >> Hsia-Jun(Randy) Li >> ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v3 10/11] media: v4l2: Add DELETE_BUF ioctl 2023-07-03 8:12 ` Benjamin Gaignard @ 2023-07-03 8:19 ` Hsia-Jun Li 2023-07-03 8:52 ` Benjamin Gaignard 0 siblings, 1 reply; 27+ messages in thread From: Hsia-Jun Li @ 2023-07-03 8:19 UTC (permalink / raw) To: Benjamin Gaignard Cc: linux-media, linux-kernel, p.zabel, ming.qian, hverkuil-cisco, gregkh, ezequiel, linux-arm-kernel, linux-mediatek, linux-arm-msm, linux-rockchip, nicolas.dufresne, linux-staging, kernel, mchehab, tfiga, m.szyprowski, ayaka On 7/3/23 16:12, Benjamin Gaignard wrote: > CAUTION: Email originated externally, do not click links or open > attachments unless you recognize the sender and know the content is safe. > > > Le 30/06/2023 à 11:43, Hsia-Jun Li a écrit : >> >> On 6/27/23 16:47, Hsia-Jun Li wrote: >>> CAUTION: Email originated externally, do not click links or open >>> attachments unless you recognize the sender and know the content is >>> safe. >>> >>> >>> On 6/27/23 16:43, Benjamin Gaignard wrote: >>>> CAUTION: Email originated externally, do not click links or open >>>> attachments unless you recognize the sender and know the content is >>>> safe. >>>> >>>> >>>> Le 27/06/2023 à 09:30, Hsia-Jun Li a écrit : >>>>> >>>>> On 6/22/23 21:13, Benjamin Gaignard wrote: >>>>>> CAUTION: Email originated externally, do not click links or open >>>>>> attachments unless you recognize the sender and know the content is >>>>>> safe. >>>>>> >>>>>> >>>>>> VIDIOC_DELETE_BUF ioctl allows to delete a buffer from a queue. >>>>>> >>>>>> Signed-off-by: Benjamin Gaignard <benjamin.gaignard@collabora.com> >>>>>> --- >>>>>> .../userspace-api/media/v4l/user-func.rst | 1 + >>>>>> .../media/v4l/vidioc-delete-buf.rst | 51 >>>>>> +++++++++++++++++++ >>>>>> .../media/common/videobuf2/videobuf2-core.c | 33 ++++++++++++ >>>>>> .../media/common/videobuf2/videobuf2-v4l2.c | 6 +++ >>>>>> drivers/media/v4l2-core/v4l2-dev.c | 1 + >>>>>> drivers/media/v4l2-core/v4l2-ioctl.c | 10 ++++ >>>>>> include/media/v4l2-ioctl.h | 4 ++ >>>>>> include/media/videobuf2-core.h | 9 ++++ >>>>>> include/media/videobuf2-v4l2.h | 11 ++++ >>>>>> include/uapi/linux/videodev2.h | 2 + >>>>>> 10 files changed, 128 insertions(+) >>>>>> create mode 100644 >>>>>> Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>> >>>>>> diff --git a/Documentation/userspace-api/media/v4l/user-func.rst >>>>>> b/Documentation/userspace-api/media/v4l/user-func.rst >>>>>> index 15ff0bf7bbe6..8c74016e12fd 100644 >>>>>> --- a/Documentation/userspace-api/media/v4l/user-func.rst >>>>>> +++ b/Documentation/userspace-api/media/v4l/user-func.rst >>>>>> @@ -17,6 +17,7 @@ Function Reference >>>>>> vidioc-dbg-g-chip-info >>>>>> vidioc-dbg-g-register >>>>>> vidioc-decoder-cmd >>>>>> + vidioc-delete-buf >>>>>> vidioc-dqevent >>>>>> vidioc-dv-timings-cap >>>>>> vidioc-encoder-cmd >>>>>> diff --git >>>>>> a/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>> b/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>> new file mode 100644 >>>>>> index 000000000000..0e7ce58f91bc >>>>>> --- /dev/null >>>>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>> @@ -0,0 +1,51 @@ >>>>>> +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later >>>>>> +.. c:namespace:: V4L >>>>>> + >>>>>> +.. _VIDIOC_DELETE_BUF: >>>>>> + >>>>>> +************************ >>>>>> +ioctl VIDIOC_DELETE_BUF >>>>>> +************************ >>>>>> + >>>>>> +Name >>>>>> +==== >>>>>> + >>>>>> +VIDIOC_DELETE_BUF - Delete a buffer from a queue >>>>>> + >>>>>> +Synopsis >>>>>> +======== >>>>>> + >>>>>> +.. c:macro:: VIDIOC_DELETE_BUF >>>>>> + >>>>>> +``int ioctl(int fd, VIDIOC_DELETE_BUF, struct v4l2_buffer *argp)`` >>>>>> + >>>>>> +Arguments >>>>>> +========= >>>>>> + >>>>>> +``fd`` >>>>>> + File descriptor returned by :c:func:`open()`. >>>>>> + >>>>>> +``argp`` >>>>>> + Pointer to struct :c:type:`v4l2_buffer`. >>>>>> + >>>>>> +Description >>>>>> +=========== >>>>>> + >>>>>> +Applications can optionally call the :ref:`VIDIOC_DELETE_BUF` >>>>>> ioctl to >>>>>> +delete a buffer from a queue. >>>>>> + >>>>>> +The struct :c:type:`v4l2_buffer` structure is specified in >>>>>> +:ref:`buffer`. >>>>>> + >>>>>> +Return Value >>>>>> +============ >>>>>> + >>>>>> +On success 0 is returned, on error -1 and the ``errno`` variable is >>>>>> set >>>>>> +appropriately. The generic error codes are described at the >>>>>> +:ref:`Generic Error Codes <gen-errors>` chapter. >>>>>> + >>>>>> +EBUSY >>>>>> + File I/O is in progress. >>>>>> + >>>>>> +EINVAL >>>>>> + The buffer ``index`` doesn't exist in the queue. >>>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c >>>>>> b/drivers/media/common/videobuf2/videobuf2-core.c >>>>>> index 899783f67580..aa546c972c3d 100644 >>>>>> --- a/drivers/media/common/videobuf2/videobuf2-core.c >>>>>> +++ b/drivers/media/common/videobuf2/videobuf2-core.c >>>>>> @@ -1637,6 +1637,39 @@ int vb2_core_prepare_buf(struct vb2_queue *q, >>>>>> unsigned int index, void *pb) >>>>>> } >>>>>> EXPORT_SYMBOL_GPL(vb2_core_prepare_buf); >>>>>> >>>>>> +int vb2_core_delete_buf(struct vb2_queue *q, unsigned int index) >>>>>> +{ >>>>>> + struct vb2_buffer *vb; >>>>>> + >>>>>> + vb = vb2_get_buffer(q, index); >>>>>> + if (!vb) { >>>>>> + dprintk(q, 1, "invalid buffer index %d\n", index); >>>>>> + return -EINVAL; >>>>>> + } >>>>>> + >>>>>> + if (vb->state != VB2_BUF_STATE_DEQUEUED) { >>>>>> + dprintk(q, 1, "can't delete non dequeued buffer >>>>>> index >>>>>> %d\n", index); >>>>>> + return -EINVAL; >>>>>> + } >>>>>> + >>>>> I know the driver could implement its own >>>>> v4l2_ioctl_ops->vidioc_delete_buf() that check whether a buffer is >>>>> used by the hardware as a future reference frame. >>>>> But I think we need a flag to let the user know which buffer is still >>>>> used by the hardware. >>>>> Alternative ref case is safe, we only know it's existing when it is >>>>> dequeued in current V4L2 buffer mechanism. >>>>> While the Golden reference frame, such long term reference frame >>>>> could >>>>> last much longer. >>>> >>>> It is up to userland stack to know frames life time, it got the >>>> information for that. >>> >>> That is true for the stateless codec driver. >>> >>> While application for stateful decoder could never do that. It also >>> breaks what the document said: >>> >>> "The backing memory of |CAPTURE| buffers that are used as reference >>> frames by the stream may be read by the hardware even after they are >>> dequeued. Consequently, the client should avoid writing into this >>> memory >>> while the |CAPTURE| queue is streaming. Failure to observe this may >>> result in corruption of decoded frames." >>> >>>> >>>>>> + if (vb->planes[0].mem_priv) >>>>>> + call_void_vb_qop(vb, buf_cleanup, vb); >>>>>> + >>>>>> + /* Free MMAP buffers or release USERPTR buffers */ >>>>>> + if (q->memory == VB2_MEMORY_MMAP) >>>>>> + __vb2_buf_mem_free(vb); >> >> Here is another problem for the existing application, the mmap() from >> the mmap offset or exportbuffer fd would not create a reference to >> buffer in this step(while the exportbuffer would create one itself). >> >> When you delete a buffer, you may not release it from its virtual >> memory space, leaving a corrupted virtual memory space. Also this >> behavior is right, because mmap(2) says: >> >> "After the mmap() call has returned, the file descriptor, fd, can >> be closed immediately without invalidating the map‐ping." > > Existing applications do not call DELETE_BUF ioctl and when call it > they will be aware that the buffer is removed. > I have done it in GStreamer: > https://urldefense.proofpoint.com/v2/url?u=https-3A__gitlab.freedesktop.org_benjamin.gaignard1_gstreamer_-2D_commit_fca0fbc934f4440693ce0ff6c8dc8a2e5f5f17d9&d=DwIDaQ&c=7dfBJ8cXbWjhc0BhImu8wVIoUFmBzj1s88r8EGyM0UY&r=P4xb2_7biqBxD4LGGPrSV6j-jf3C3xlR7PXU-mLTeZE&m=TGH9toTzGRfO5aBsfaMvGbcOw-28q6cPmpX6vScbHjpCtaLtb-RuvBvsJ0z9RvAB&s=Ufl1ccfRZf2EhnfCBvnQzRJV9CDhGxl5spe9WNECspU&e= > I have read that. There is not a VP8 parser in Gstreamer, while a parser would not work when deal with the secure video(userspace can't access the data context at all). Besides, this adds extra work for the application for a stateful codec driver. The application need to parser the bitstream and track the dpb. I don't mind if you could fix the nonfiction mechanism for those non-display frame and internal reference state. That could be requirement for codec firmware that its driver could support this DELETE_BUF ioctl() feature. > > Regards, > Benjamin > >> >>>>>> + else if (q->memory == VB2_MEMORY_DMABUF) >>>>>> + __vb2_buf_dmabuf_put(vb); >>>>>> + else >>>>>> + __vb2_buf_userptr_put(vb); >>>>>> + >>>>>> + vb2_queue_remove_buffer(q, vb); >>>>>> + kfree(vb); >>>>>> + >>>>>> + dprintk(q, 2, "buffer %d deleted\n", index); >>>>>> + return 0; >>>>>> +} >>>>>> + >>>>>> /* >>>>>> * vb2_start_streaming() - Attempt to start streaming. >>>>>> * @q: videobuf2 queue >>>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>> b/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>> index 724135d41f7f..cea666c17b41 100644 >>>>>> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>> @@ -751,6 +751,12 @@ int vb2_prepare_buf(struct vb2_queue *q, struct >>>>>> media_device *mdev, >>>>>> } >>>>>> EXPORT_SYMBOL_GPL(vb2_prepare_buf); >>>>>> >>>>>> +int vb2_delete_buf(struct vb2_queue *q, struct v4l2_buffer *b) >>>>>> +{ >>>>>> + return vb2_core_delete_buf(q, b->index); >>>>>> +} >>>>>> +EXPORT_SYMBOL_GPL(vb2_delete_buf); >>>>>> + >>>>>> int vb2_create_bufs(struct vb2_queue *q, struct >>>>>> v4l2_create_buffers >>>>>> *create) >>>>>> { >>>>>> unsigned requested_planes = 1; >>>>>> diff --git a/drivers/media/v4l2-core/v4l2-dev.c >>>>>> b/drivers/media/v4l2-core/v4l2-dev.c >>>>>> index f81279492682..80ace2e1e932 100644 >>>>>> --- a/drivers/media/v4l2-core/v4l2-dev.c >>>>>> +++ b/drivers/media/v4l2-core/v4l2-dev.c >>>>>> @@ -720,6 +720,7 @@ static void determine_valid_ioctls(struct >>>>>> video_device *vdev) >>>>>> SET_VALID_IOCTL(ops, VIDIOC_PREPARE_BUF, >>>>>> vidioc_prepare_buf); >>>>>> SET_VALID_IOCTL(ops, VIDIOC_STREAMON, >>>>>> vidioc_streamon); >>>>>> SET_VALID_IOCTL(ops, VIDIOC_STREAMOFF, >>>>>> vidioc_streamoff); >>>>>> + SET_VALID_IOCTL(ops, VIDIOC_DELETE_BUF, >>>>>> vidioc_delete_buf); >>>>>> } >>>>>> >>>>>> if (is_vid || is_vbi || is_meta) { >>>>>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>> b/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>> index a858acea6547..1c737279d3ef 100644 >>>>>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>> @@ -2156,6 +2156,15 @@ static int v4l_prepare_buf(const struct >>>>>> v4l2_ioctl_ops *ops, >>>>>> return ret ? ret : ops->vidioc_prepare_buf(file, fh, b); >>>>>> } >>>>>> >>>>>> +static int v4l_delete_buf(const struct v4l2_ioctl_ops *ops, >>>>>> + struct file *file, void *fh, void *arg) >>>>>> +{ >>>>>> + struct v4l2_buffer *b = arg; >>>>>> + int ret = check_fmt(file, b->type); >>>>>> + >>>>>> + return ret ? ret : ops->vidioc_delete_buf(file, fh, b); >>>>>> +} >>>>>> + >>>>>> static int v4l_g_parm(const struct v4l2_ioctl_ops *ops, >>>>>> struct file *file, void *fh, void >>>>>> *arg) >>>>>> { >>>>>> @@ -2905,6 +2914,7 @@ static const struct v4l2_ioctl_info >>>>>> v4l2_ioctls[] = { >>>>>> IOCTL_INFO(VIDIOC_ENUM_FREQ_BANDS, v4l_enum_freq_bands, >>>>>> v4l_print_freq_band, 0), >>>>>> IOCTL_INFO(VIDIOC_DBG_G_CHIP_INFO, v4l_dbg_g_chip_info, >>>>>> v4l_print_dbg_chip_info, INFO_FL_CLEAR(v4l2_dbg_chip_info, match)), >>>>>> IOCTL_INFO(VIDIOC_QUERY_EXT_CTRL, v4l_query_ext_ctrl, >>>>>> v4l_print_query_ext_ctrl, INFO_FL_CTRL | >>>>>> INFO_FL_CLEAR(v4l2_query_ext_ctrl, id)), >>>>>> + IOCTL_INFO(VIDIOC_DELETE_BUF, v4l_delete_buf, >>>>>> v4l_print_buffer, INFO_FL_QUEUE), >>>>>> }; >>>>>> #define V4L2_IOCTLS ARRAY_SIZE(v4l2_ioctls) >>>>>> >>>>>> diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h >>>>>> index edb733f21604..2f232ed884c7 100644 >>>>>> --- a/include/media/v4l2-ioctl.h >>>>>> +++ b/include/media/v4l2-ioctl.h >>>>>> @@ -163,6 +163,8 @@ struct v4l2_fh; >>>>>> * :ref:`VIDIOC_CREATE_BUFS <vidioc_create_bufs>` ioctl >>>>>> * @vidioc_prepare_buf: pointer to the function that implements >>>>>> * :ref:`VIDIOC_PREPARE_BUF <vidioc_prepare_buf>` ioctl >>>>>> + * @vidioc_delete_buf: pointer to the function that implements >>>>>> + * :ref:`VIDIOC_DELETE_BUF <vidioc_delete_buf>` ioctl >>>>>> * @vidioc_overlay: pointer to the function that implements >>>>>> * :ref:`VIDIOC_OVERLAY <vidioc_overlay>` ioctl >>>>>> * @vidioc_g_fbuf: pointer to the function that implements >>>>>> @@ -422,6 +424,8 @@ struct v4l2_ioctl_ops { >>>>>> struct v4l2_create_buffers *b); >>>>>> int (*vidioc_prepare_buf)(struct file *file, void *fh, >>>>>> struct v4l2_buffer *b); >>>>>> + int (*vidioc_delete_buf)(struct file *file, void *fh, >>>>>> + struct v4l2_buffer *b); >>>>>> >>>>>> int (*vidioc_overlay)(struct file *file, void *fh, unsigned >>>>>> int i); >>>>>> int (*vidioc_g_fbuf)(struct file *file, void *fh, >>>>>> diff --git a/include/media/videobuf2-core.h >>>>>> b/include/media/videobuf2-core.h >>>>>> index 080b783d608d..0f9e68f76b77 100644 >>>>>> --- a/include/media/videobuf2-core.h >>>>>> +++ b/include/media/videobuf2-core.h >>>>>> @@ -840,6 +840,15 @@ int vb2_core_create_bufs(struct vb2_queue *q, >>>>>> enum vb2_memory memory, >>>>>> */ >>>>>> int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, >>>>>> void *pb); >>>>>> >>>>>> +/** >>>>>> + * vb2_core_delete_buf() - >>>>>> + * @q: pointer to &struct vb2_queue with videobuf2 queue. >>>>>> + * @index: id number of the buffer. >>>>>> + * >>>>>> + * Return: returns zero on success; an error code otherwise. >>>>>> + */ >>>>>> +int vb2_core_delete_buf(struct vb2_queue *q, unsigned int index); >>>>>> + >>>>>> /** >>>>>> * vb2_core_qbuf() - Queue a buffer from userspace >>>>>> * >>>>>> diff --git a/include/media/videobuf2-v4l2.h >>>>>> b/include/media/videobuf2-v4l2.h >>>>>> index 88a7a565170e..3beeb4c735f0 100644 >>>>>> --- a/include/media/videobuf2-v4l2.h >>>>>> +++ b/include/media/videobuf2-v4l2.h >>>>>> @@ -114,6 +114,17 @@ int vb2_create_bufs(struct vb2_queue *q, struct >>>>>> v4l2_create_buffers *create); >>>>>> */ >>>>>> int vb2_prepare_buf(struct vb2_queue *q, struct media_device >>>>>> *mdev, >>>>>> struct v4l2_buffer *b); >>>>>> +/** >>>>>> + * vb2_delete_buf() - Delete the buffer from the queue >>>>>> + * >>>>>> + * @q: pointer to &struct vb2_queue with videobuf2 queue. >>>>>> + * @b: buffer structure passed from userspace to >>>>>> + * &v4l2_ioctl_ops->vidioc_delete_buf handler in driver >>>>>> + * >>>>>> + * The return values from this function are intended to be directly >>>>>> returned >>>>>> + * from &v4l2_ioctl_ops->vidioc_delete_buf handler in driver. >>>>>> + */ >>>>>> +int vb2_delete_buf(struct vb2_queue *q, struct v4l2_buffer *b); >>>>>> >>>>>> /** >>>>>> * vb2_qbuf() - Queue a buffer from userspace >>>>>> diff --git a/include/uapi/linux/videodev2.h >>>>>> b/include/uapi/linux/videodev2.h >>>>>> index aee75eb9e686..31bba1915642 100644 >>>>>> --- a/include/uapi/linux/videodev2.h >>>>>> +++ b/include/uapi/linux/videodev2.h >>>>>> @@ -2702,6 +2702,8 @@ struct v4l2_create_buffers { >>>>>> #define VIDIOC_DBG_G_CHIP_INFO _IOWR('V', 102, struct >>>>>> v4l2_dbg_chip_info) >>>>>> >>>>>> #define VIDIOC_QUERY_EXT_CTRL _IOWR('V', 103, struct >>>>>> v4l2_query_ext_ctrl) >>>>>> +#define VIDIOC_DELETE_BUF _IOWR('V', 104, struct v4l2_buffer) >>>>>> + >>>>>> >>>>>> /* Reminder: when adding new ioctls please add support for them to >>>>>> drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */ >>>>>> -- >>>>>> 2.39.2 >>>>>> >>> -- >>> Hsia-Jun(Randy) Li >>> -- Hsia-Jun(Randy) Li ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v3 10/11] media: v4l2: Add DELETE_BUF ioctl 2023-07-03 8:19 ` Hsia-Jun Li @ 2023-07-03 8:52 ` Benjamin Gaignard 2023-07-03 9:20 ` Hsia-Jun Li 0 siblings, 1 reply; 27+ messages in thread From: Benjamin Gaignard @ 2023-07-03 8:52 UTC (permalink / raw) To: Hsia-Jun Li Cc: linux-media, linux-kernel, p.zabel, ming.qian, hverkuil-cisco, gregkh, ezequiel, linux-arm-kernel, linux-mediatek, linux-arm-msm, linux-rockchip, nicolas.dufresne, linux-staging, kernel, mchehab, tfiga, m.szyprowski, ayaka Le 03/07/2023 à 10:19, Hsia-Jun Li a écrit : > > On 7/3/23 16:12, Benjamin Gaignard wrote: >> CAUTION: Email originated externally, do not click links or open >> attachments unless you recognize the sender and know the content is >> safe. >> >> >> Le 30/06/2023 à 11:43, Hsia-Jun Li a écrit : >>> >>> On 6/27/23 16:47, Hsia-Jun Li wrote: >>>> CAUTION: Email originated externally, do not click links or open >>>> attachments unless you recognize the sender and know the content is >>>> safe. >>>> >>>> >>>> On 6/27/23 16:43, Benjamin Gaignard wrote: >>>>> CAUTION: Email originated externally, do not click links or open >>>>> attachments unless you recognize the sender and know the content is >>>>> safe. >>>>> >>>>> >>>>> Le 27/06/2023 à 09:30, Hsia-Jun Li a écrit : >>>>>> >>>>>> On 6/22/23 21:13, Benjamin Gaignard wrote: >>>>>>> CAUTION: Email originated externally, do not click links or open >>>>>>> attachments unless you recognize the sender and know the content is >>>>>>> safe. >>>>>>> >>>>>>> >>>>>>> VIDIOC_DELETE_BUF ioctl allows to delete a buffer from a queue. >>>>>>> >>>>>>> Signed-off-by: Benjamin Gaignard <benjamin.gaignard@collabora.com> >>>>>>> --- >>>>>>> .../userspace-api/media/v4l/user-func.rst | 1 + >>>>>>> .../media/v4l/vidioc-delete-buf.rst | 51 >>>>>>> +++++++++++++++++++ >>>>>>> .../media/common/videobuf2/videobuf2-core.c | 33 ++++++++++++ >>>>>>> .../media/common/videobuf2/videobuf2-v4l2.c | 6 +++ >>>>>>> drivers/media/v4l2-core/v4l2-dev.c | 1 + >>>>>>> drivers/media/v4l2-core/v4l2-ioctl.c | 10 ++++ >>>>>>> include/media/v4l2-ioctl.h | 4 ++ >>>>>>> include/media/videobuf2-core.h | 9 ++++ >>>>>>> include/media/videobuf2-v4l2.h | 11 ++++ >>>>>>> include/uapi/linux/videodev2.h | 2 + >>>>>>> 10 files changed, 128 insertions(+) >>>>>>> create mode 100644 >>>>>>> Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>>> >>>>>>> diff --git a/Documentation/userspace-api/media/v4l/user-func.rst >>>>>>> b/Documentation/userspace-api/media/v4l/user-func.rst >>>>>>> index 15ff0bf7bbe6..8c74016e12fd 100644 >>>>>>> --- a/Documentation/userspace-api/media/v4l/user-func.rst >>>>>>> +++ b/Documentation/userspace-api/media/v4l/user-func.rst >>>>>>> @@ -17,6 +17,7 @@ Function Reference >>>>>>> vidioc-dbg-g-chip-info >>>>>>> vidioc-dbg-g-register >>>>>>> vidioc-decoder-cmd >>>>>>> + vidioc-delete-buf >>>>>>> vidioc-dqevent >>>>>>> vidioc-dv-timings-cap >>>>>>> vidioc-encoder-cmd >>>>>>> diff --git >>>>>>> a/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>>> b/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>>> new file mode 100644 >>>>>>> index 000000000000..0e7ce58f91bc >>>>>>> --- /dev/null >>>>>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>>> @@ -0,0 +1,51 @@ >>>>>>> +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later >>>>>>> +.. c:namespace:: V4L >>>>>>> + >>>>>>> +.. _VIDIOC_DELETE_BUF: >>>>>>> + >>>>>>> +************************ >>>>>>> +ioctl VIDIOC_DELETE_BUF >>>>>>> +************************ >>>>>>> + >>>>>>> +Name >>>>>>> +==== >>>>>>> + >>>>>>> +VIDIOC_DELETE_BUF - Delete a buffer from a queue >>>>>>> + >>>>>>> +Synopsis >>>>>>> +======== >>>>>>> + >>>>>>> +.. c:macro:: VIDIOC_DELETE_BUF >>>>>>> + >>>>>>> +``int ioctl(int fd, VIDIOC_DELETE_BUF, struct v4l2_buffer *argp)`` >>>>>>> + >>>>>>> +Arguments >>>>>>> +========= >>>>>>> + >>>>>>> +``fd`` >>>>>>> + File descriptor returned by :c:func:`open()`. >>>>>>> + >>>>>>> +``argp`` >>>>>>> + Pointer to struct :c:type:`v4l2_buffer`. >>>>>>> + >>>>>>> +Description >>>>>>> +=========== >>>>>>> + >>>>>>> +Applications can optionally call the :ref:`VIDIOC_DELETE_BUF` >>>>>>> ioctl to >>>>>>> +delete a buffer from a queue. >>>>>>> + >>>>>>> +The struct :c:type:`v4l2_buffer` structure is specified in >>>>>>> +:ref:`buffer`. >>>>>>> + >>>>>>> +Return Value >>>>>>> +============ >>>>>>> + >>>>>>> +On success 0 is returned, on error -1 and the ``errno`` >>>>>>> variable is >>>>>>> set >>>>>>> +appropriately. The generic error codes are described at the >>>>>>> +:ref:`Generic Error Codes <gen-errors>` chapter. >>>>>>> + >>>>>>> +EBUSY >>>>>>> + File I/O is in progress. >>>>>>> + >>>>>>> +EINVAL >>>>>>> + The buffer ``index`` doesn't exist in the queue. >>>>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>> b/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>> index 899783f67580..aa546c972c3d 100644 >>>>>>> --- a/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>> +++ b/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>> @@ -1637,6 +1637,39 @@ int vb2_core_prepare_buf(struct vb2_queue >>>>>>> *q, >>>>>>> unsigned int index, void *pb) >>>>>>> } >>>>>>> EXPORT_SYMBOL_GPL(vb2_core_prepare_buf); >>>>>>> >>>>>>> +int vb2_core_delete_buf(struct vb2_queue *q, unsigned int index) >>>>>>> +{ >>>>>>> + struct vb2_buffer *vb; >>>>>>> + >>>>>>> + vb = vb2_get_buffer(q, index); >>>>>>> + if (!vb) { >>>>>>> + dprintk(q, 1, "invalid buffer index %d\n", index); >>>>>>> + return -EINVAL; >>>>>>> + } >>>>>>> + >>>>>>> + if (vb->state != VB2_BUF_STATE_DEQUEUED) { >>>>>>> + dprintk(q, 1, "can't delete non dequeued buffer >>>>>>> index >>>>>>> %d\n", index); >>>>>>> + return -EINVAL; >>>>>>> + } >>>>>>> + >>>>>> I know the driver could implement its own >>>>>> v4l2_ioctl_ops->vidioc_delete_buf() that check whether a buffer is >>>>>> used by the hardware as a future reference frame. >>>>>> But I think we need a flag to let the user know which buffer is >>>>>> still >>>>>> used by the hardware. >>>>>> Alternative ref case is safe, we only know it's existing when it is >>>>>> dequeued in current V4L2 buffer mechanism. >>>>>> While the Golden reference frame, such long term reference frame >>>>>> could >>>>>> last much longer. >>>>> >>>>> It is up to userland stack to know frames life time, it got the >>>>> information for that. >>>> >>>> That is true for the stateless codec driver. >>>> >>>> While application for stateful decoder could never do that. It also >>>> breaks what the document said: >>>> >>>> "The backing memory of |CAPTURE| buffers that are used as reference >>>> frames by the stream may be read by the hardware even after they are >>>> dequeued. Consequently, the client should avoid writing into this >>>> memory >>>> while the |CAPTURE| queue is streaming. Failure to observe this may >>>> result in corruption of decoded frames." >>>> >>>>> >>>>>>> + if (vb->planes[0].mem_priv) >>>>>>> + call_void_vb_qop(vb, buf_cleanup, vb); >>>>>>> + >>>>>>> + /* Free MMAP buffers or release USERPTR buffers */ >>>>>>> + if (q->memory == VB2_MEMORY_MMAP) >>>>>>> + __vb2_buf_mem_free(vb); >>> >>> Here is another problem for the existing application, the mmap() from >>> the mmap offset or exportbuffer fd would not create a reference to >>> buffer in this step(while the exportbuffer would create one itself). >>> >>> When you delete a buffer, you may not release it from its virtual >>> memory space, leaving a corrupted virtual memory space. Also this >>> behavior is right, because mmap(2) says: >>> >>> "After the mmap() call has returned, the file descriptor, fd, can >>> be closed immediately without invalidating the map‐ping." >> >> Existing applications do not call DELETE_BUF ioctl and when call it >> they will be aware that the buffer is removed. >> I have done it in GStreamer: >> https://urldefense.proofpoint.com/v2/url?u=https-3A__gitlab.freedesktop.org_benjamin.gaignard1_gstreamer_-2D_commit_fca0fbc934f4440693ce0ff6c8dc8a2e5f5f17d9&d=DwIDaQ&c=7dfBJ8cXbWjhc0BhImu8wVIoUFmBzj1s88r8EGyM0UY&r=P4xb2_7biqBxD4LGGPrSV6j-jf3C3xlR7PXU-mLTeZE&m=TGH9toTzGRfO5aBsfaMvGbcOw-28q6cPmpX6vScbHjpCtaLtb-RuvBvsJ0z9RvAB&s=Ufl1ccfRZf2EhnfCBvnQzRJV9CDhGxl5spe9WNECspU&e= >> > > I have read that. > > There is not a VP8 parser in Gstreamer, while a parser would not work > when deal with the secure video(userspace can't access the data > context at all). > > Besides, this adds extra work for the application for a stateful codec > driver. The application need to parser the bitstream and track the dpb. > > I don't mind if you could fix the nonfiction mechanism for those > non-display frame and internal reference state. > > That could be requirement for codec firmware that its driver could > support this DELETE_BUF ioctl() feature. Sorry I don't see the link with my patches here... I have work on non-secure VP9 on stateless codec. DELETE_BUF ioctl is optional and the main goal is to offer a way to applications to save memory if they know when they could delete buffers without risk. > >> >> Regards, >> Benjamin >> >>> >>>>>>> + else if (q->memory == VB2_MEMORY_DMABUF) >>>>>>> + __vb2_buf_dmabuf_put(vb); >>>>>>> + else >>>>>>> + __vb2_buf_userptr_put(vb); >>>>>>> + >>>>>>> + vb2_queue_remove_buffer(q, vb); >>>>>>> + kfree(vb); >>>>>>> + >>>>>>> + dprintk(q, 2, "buffer %d deleted\n", index); >>>>>>> + return 0; >>>>>>> +} >>>>>>> + >>>>>>> /* >>>>>>> * vb2_start_streaming() - Attempt to start streaming. >>>>>>> * @q: videobuf2 queue >>>>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>>> b/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>>> index 724135d41f7f..cea666c17b41 100644 >>>>>>> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>>> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>>> @@ -751,6 +751,12 @@ int vb2_prepare_buf(struct vb2_queue *q, >>>>>>> struct >>>>>>> media_device *mdev, >>>>>>> } >>>>>>> EXPORT_SYMBOL_GPL(vb2_prepare_buf); >>>>>>> >>>>>>> +int vb2_delete_buf(struct vb2_queue *q, struct v4l2_buffer *b) >>>>>>> +{ >>>>>>> + return vb2_core_delete_buf(q, b->index); >>>>>>> +} >>>>>>> +EXPORT_SYMBOL_GPL(vb2_delete_buf); >>>>>>> + >>>>>>> int vb2_create_bufs(struct vb2_queue *q, struct >>>>>>> v4l2_create_buffers >>>>>>> *create) >>>>>>> { >>>>>>> unsigned requested_planes = 1; >>>>>>> diff --git a/drivers/media/v4l2-core/v4l2-dev.c >>>>>>> b/drivers/media/v4l2-core/v4l2-dev.c >>>>>>> index f81279492682..80ace2e1e932 100644 >>>>>>> --- a/drivers/media/v4l2-core/v4l2-dev.c >>>>>>> +++ b/drivers/media/v4l2-core/v4l2-dev.c >>>>>>> @@ -720,6 +720,7 @@ static void determine_valid_ioctls(struct >>>>>>> video_device *vdev) >>>>>>> SET_VALID_IOCTL(ops, VIDIOC_PREPARE_BUF, >>>>>>> vidioc_prepare_buf); >>>>>>> SET_VALID_IOCTL(ops, VIDIOC_STREAMON, >>>>>>> vidioc_streamon); >>>>>>> SET_VALID_IOCTL(ops, VIDIOC_STREAMOFF, >>>>>>> vidioc_streamoff); >>>>>>> + SET_VALID_IOCTL(ops, VIDIOC_DELETE_BUF, >>>>>>> vidioc_delete_buf); >>>>>>> } >>>>>>> >>>>>>> if (is_vid || is_vbi || is_meta) { >>>>>>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>>> b/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>>> index a858acea6547..1c737279d3ef 100644 >>>>>>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>>> @@ -2156,6 +2156,15 @@ static int v4l_prepare_buf(const struct >>>>>>> v4l2_ioctl_ops *ops, >>>>>>> return ret ? ret : ops->vidioc_prepare_buf(file, fh, b); >>>>>>> } >>>>>>> >>>>>>> +static int v4l_delete_buf(const struct v4l2_ioctl_ops *ops, >>>>>>> + struct file *file, void *fh, void *arg) >>>>>>> +{ >>>>>>> + struct v4l2_buffer *b = arg; >>>>>>> + int ret = check_fmt(file, b->type); >>>>>>> + >>>>>>> + return ret ? ret : ops->vidioc_delete_buf(file, fh, b); >>>>>>> +} >>>>>>> + >>>>>>> static int v4l_g_parm(const struct v4l2_ioctl_ops *ops, >>>>>>> struct file *file, void *fh, void >>>>>>> *arg) >>>>>>> { >>>>>>> @@ -2905,6 +2914,7 @@ static const struct v4l2_ioctl_info >>>>>>> v4l2_ioctls[] = { >>>>>>> IOCTL_INFO(VIDIOC_ENUM_FREQ_BANDS, v4l_enum_freq_bands, >>>>>>> v4l_print_freq_band, 0), >>>>>>> IOCTL_INFO(VIDIOC_DBG_G_CHIP_INFO, v4l_dbg_g_chip_info, >>>>>>> v4l_print_dbg_chip_info, INFO_FL_CLEAR(v4l2_dbg_chip_info, match)), >>>>>>> IOCTL_INFO(VIDIOC_QUERY_EXT_CTRL, v4l_query_ext_ctrl, >>>>>>> v4l_print_query_ext_ctrl, INFO_FL_CTRL | >>>>>>> INFO_FL_CLEAR(v4l2_query_ext_ctrl, id)), >>>>>>> + IOCTL_INFO(VIDIOC_DELETE_BUF, v4l_delete_buf, >>>>>>> v4l_print_buffer, INFO_FL_QUEUE), >>>>>>> }; >>>>>>> #define V4L2_IOCTLS ARRAY_SIZE(v4l2_ioctls) >>>>>>> >>>>>>> diff --git a/include/media/v4l2-ioctl.h >>>>>>> b/include/media/v4l2-ioctl.h >>>>>>> index edb733f21604..2f232ed884c7 100644 >>>>>>> --- a/include/media/v4l2-ioctl.h >>>>>>> +++ b/include/media/v4l2-ioctl.h >>>>>>> @@ -163,6 +163,8 @@ struct v4l2_fh; >>>>>>> * :ref:`VIDIOC_CREATE_BUFS <vidioc_create_bufs>` ioctl >>>>>>> * @vidioc_prepare_buf: pointer to the function that implements >>>>>>> * :ref:`VIDIOC_PREPARE_BUF <vidioc_prepare_buf>` ioctl >>>>>>> + * @vidioc_delete_buf: pointer to the function that implements >>>>>>> + * :ref:`VIDIOC_DELETE_BUF <vidioc_delete_buf>` ioctl >>>>>>> * @vidioc_overlay: pointer to the function that implements >>>>>>> * :ref:`VIDIOC_OVERLAY <vidioc_overlay>` ioctl >>>>>>> * @vidioc_g_fbuf: pointer to the function that implements >>>>>>> @@ -422,6 +424,8 @@ struct v4l2_ioctl_ops { >>>>>>> struct v4l2_create_buffers *b); >>>>>>> int (*vidioc_prepare_buf)(struct file *file, void *fh, >>>>>>> struct v4l2_buffer *b); >>>>>>> + int (*vidioc_delete_buf)(struct file *file, void *fh, >>>>>>> + struct v4l2_buffer *b); >>>>>>> >>>>>>> int (*vidioc_overlay)(struct file *file, void *fh, >>>>>>> unsigned >>>>>>> int i); >>>>>>> int (*vidioc_g_fbuf)(struct file *file, void *fh, >>>>>>> diff --git a/include/media/videobuf2-core.h >>>>>>> b/include/media/videobuf2-core.h >>>>>>> index 080b783d608d..0f9e68f76b77 100644 >>>>>>> --- a/include/media/videobuf2-core.h >>>>>>> +++ b/include/media/videobuf2-core.h >>>>>>> @@ -840,6 +840,15 @@ int vb2_core_create_bufs(struct vb2_queue *q, >>>>>>> enum vb2_memory memory, >>>>>>> */ >>>>>>> int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, >>>>>>> void *pb); >>>>>>> >>>>>>> +/** >>>>>>> + * vb2_core_delete_buf() - >>>>>>> + * @q: pointer to &struct vb2_queue with videobuf2 queue. >>>>>>> + * @index: id number of the buffer. >>>>>>> + * >>>>>>> + * Return: returns zero on success; an error code otherwise. >>>>>>> + */ >>>>>>> +int vb2_core_delete_buf(struct vb2_queue *q, unsigned int index); >>>>>>> + >>>>>>> /** >>>>>>> * vb2_core_qbuf() - Queue a buffer from userspace >>>>>>> * >>>>>>> diff --git a/include/media/videobuf2-v4l2.h >>>>>>> b/include/media/videobuf2-v4l2.h >>>>>>> index 88a7a565170e..3beeb4c735f0 100644 >>>>>>> --- a/include/media/videobuf2-v4l2.h >>>>>>> +++ b/include/media/videobuf2-v4l2.h >>>>>>> @@ -114,6 +114,17 @@ int vb2_create_bufs(struct vb2_queue *q, >>>>>>> struct >>>>>>> v4l2_create_buffers *create); >>>>>>> */ >>>>>>> int vb2_prepare_buf(struct vb2_queue *q, struct media_device >>>>>>> *mdev, >>>>>>> struct v4l2_buffer *b); >>>>>>> +/** >>>>>>> + * vb2_delete_buf() - Delete the buffer from the queue >>>>>>> + * >>>>>>> + * @q: pointer to &struct vb2_queue with videobuf2 queue. >>>>>>> + * @b: buffer structure passed from userspace to >>>>>>> + * &v4l2_ioctl_ops->vidioc_delete_buf handler in driver >>>>>>> + * >>>>>>> + * The return values from this function are intended to be >>>>>>> directly >>>>>>> returned >>>>>>> + * from &v4l2_ioctl_ops->vidioc_delete_buf handler in driver. >>>>>>> + */ >>>>>>> +int vb2_delete_buf(struct vb2_queue *q, struct v4l2_buffer *b); >>>>>>> >>>>>>> /** >>>>>>> * vb2_qbuf() - Queue a buffer from userspace >>>>>>> diff --git a/include/uapi/linux/videodev2.h >>>>>>> b/include/uapi/linux/videodev2.h >>>>>>> index aee75eb9e686..31bba1915642 100644 >>>>>>> --- a/include/uapi/linux/videodev2.h >>>>>>> +++ b/include/uapi/linux/videodev2.h >>>>>>> @@ -2702,6 +2702,8 @@ struct v4l2_create_buffers { >>>>>>> #define VIDIOC_DBG_G_CHIP_INFO _IOWR('V', 102, struct >>>>>>> v4l2_dbg_chip_info) >>>>>>> >>>>>>> #define VIDIOC_QUERY_EXT_CTRL _IOWR('V', 103, struct >>>>>>> v4l2_query_ext_ctrl) >>>>>>> +#define VIDIOC_DELETE_BUF _IOWR('V', 104, struct v4l2_buffer) >>>>>>> + >>>>>>> >>>>>>> /* Reminder: when adding new ioctls please add support for >>>>>>> them to >>>>>>> drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */ >>>>>>> -- >>>>>>> 2.39.2 >>>>>>> >>>> -- >>>> Hsia-Jun(Randy) Li >>>> ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v3 10/11] media: v4l2: Add DELETE_BUF ioctl 2023-07-03 8:52 ` Benjamin Gaignard @ 2023-07-03 9:20 ` Hsia-Jun Li 2023-07-03 10:35 ` Benjamin Gaignard 0 siblings, 1 reply; 27+ messages in thread From: Hsia-Jun Li @ 2023-07-03 9:20 UTC (permalink / raw) To: Benjamin Gaignard Cc: linux-media, linux-kernel, p.zabel, ming.qian, hverkuil-cisco, gregkh, ezequiel, linux-arm-kernel, linux-mediatek, linux-arm-msm, linux-rockchip, nicolas.dufresne, linux-staging, kernel, mchehab, tfiga, m.szyprowski, ayaka On 7/3/23 16:52, Benjamin Gaignard wrote: > CAUTION: Email originated externally, do not click links or open > attachments unless you recognize the sender and know the content is safe. > > > Le 03/07/2023 à 10:19, Hsia-Jun Li a écrit : >> >> On 7/3/23 16:12, Benjamin Gaignard wrote: >>> CAUTION: Email originated externally, do not click links or open >>> attachments unless you recognize the sender and know the content is >>> safe. >>> >>> >>> Le 30/06/2023 à 11:43, Hsia-Jun Li a écrit : >>>> >>>> On 6/27/23 16:47, Hsia-Jun Li wrote: >>>>> CAUTION: Email originated externally, do not click links or open >>>>> attachments unless you recognize the sender and know the content is >>>>> safe. >>>>> >>>>> >>>>> On 6/27/23 16:43, Benjamin Gaignard wrote: >>>>>> CAUTION: Email originated externally, do not click links or open >>>>>> attachments unless you recognize the sender and know the content is >>>>>> safe. >>>>>> >>>>>> >>>>>> Le 27/06/2023 à 09:30, Hsia-Jun Li a écrit : >>>>>>> >>>>>>> On 6/22/23 21:13, Benjamin Gaignard wrote: >>>>>>>> CAUTION: Email originated externally, do not click links or open >>>>>>>> attachments unless you recognize the sender and know the >>>>>>>> content is >>>>>>>> safe. >>>>>>>> >>>>>>>> >>>>>>>> VIDIOC_DELETE_BUF ioctl allows to delete a buffer from a queue. >>>>>>>> >>>>>>>> Signed-off-by: Benjamin Gaignard <benjamin.gaignard@collabora.com> >>>>>>>> --- >>>>>>>> .../userspace-api/media/v4l/user-func.rst | 1 + >>>>>>>> .../media/v4l/vidioc-delete-buf.rst | 51 >>>>>>>> +++++++++++++++++++ >>>>>>>> .../media/common/videobuf2/videobuf2-core.c | 33 ++++++++++++ >>>>>>>> .../media/common/videobuf2/videobuf2-v4l2.c | 6 +++ >>>>>>>> drivers/media/v4l2-core/v4l2-dev.c | 1 + >>>>>>>> drivers/media/v4l2-core/v4l2-ioctl.c | 10 ++++ >>>>>>>> include/media/v4l2-ioctl.h | 4 ++ >>>>>>>> include/media/videobuf2-core.h | 9 ++++ >>>>>>>> include/media/videobuf2-v4l2.h | 11 ++++ >>>>>>>> include/uapi/linux/videodev2.h | 2 + >>>>>>>> 10 files changed, 128 insertions(+) >>>>>>>> create mode 100644 >>>>>>>> Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>>>> >>>>>>>> diff --git a/Documentation/userspace-api/media/v4l/user-func.rst >>>>>>>> b/Documentation/userspace-api/media/v4l/user-func.rst >>>>>>>> index 15ff0bf7bbe6..8c74016e12fd 100644 >>>>>>>> --- a/Documentation/userspace-api/media/v4l/user-func.rst >>>>>>>> +++ b/Documentation/userspace-api/media/v4l/user-func.rst >>>>>>>> @@ -17,6 +17,7 @@ Function Reference >>>>>>>> vidioc-dbg-g-chip-info >>>>>>>> vidioc-dbg-g-register >>>>>>>> vidioc-decoder-cmd >>>>>>>> + vidioc-delete-buf >>>>>>>> vidioc-dqevent >>>>>>>> vidioc-dv-timings-cap >>>>>>>> vidioc-encoder-cmd >>>>>>>> diff --git >>>>>>>> a/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>>>> b/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>>>> new file mode 100644 >>>>>>>> index 000000000000..0e7ce58f91bc >>>>>>>> --- /dev/null >>>>>>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>>>> @@ -0,0 +1,51 @@ >>>>>>>> +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later >>>>>>>> +.. c:namespace:: V4L >>>>>>>> + >>>>>>>> +.. _VIDIOC_DELETE_BUF: >>>>>>>> + >>>>>>>> +************************ >>>>>>>> +ioctl VIDIOC_DELETE_BUF >>>>>>>> +************************ >>>>>>>> + >>>>>>>> +Name >>>>>>>> +==== >>>>>>>> + >>>>>>>> +VIDIOC_DELETE_BUF - Delete a buffer from a queue >>>>>>>> + >>>>>>>> +Synopsis >>>>>>>> +======== >>>>>>>> + >>>>>>>> +.. c:macro:: VIDIOC_DELETE_BUF >>>>>>>> + >>>>>>>> +``int ioctl(int fd, VIDIOC_DELETE_BUF, struct v4l2_buffer >>>>>>>> *argp)`` >>>>>>>> + >>>>>>>> +Arguments >>>>>>>> +========= >>>>>>>> + >>>>>>>> +``fd`` >>>>>>>> + File descriptor returned by :c:func:`open()`. >>>>>>>> + >>>>>>>> +``argp`` >>>>>>>> + Pointer to struct :c:type:`v4l2_buffer`. >>>>>>>> + >>>>>>>> +Description >>>>>>>> +=========== >>>>>>>> + >>>>>>>> +Applications can optionally call the :ref:`VIDIOC_DELETE_BUF` >>>>>>>> ioctl to >>>>>>>> +delete a buffer from a queue. >>>>>>>> + >>>>>>>> +The struct :c:type:`v4l2_buffer` structure is specified in >>>>>>>> +:ref:`buffer`. >>>>>>>> + >>>>>>>> +Return Value >>>>>>>> +============ >>>>>>>> + >>>>>>>> +On success 0 is returned, on error -1 and the ``errno`` >>>>>>>> variable is >>>>>>>> set >>>>>>>> +appropriately. The generic error codes are described at the >>>>>>>> +:ref:`Generic Error Codes <gen-errors>` chapter. >>>>>>>> + >>>>>>>> +EBUSY >>>>>>>> + File I/O is in progress. >>>>>>>> + >>>>>>>> +EINVAL >>>>>>>> + The buffer ``index`` doesn't exist in the queue. >>>>>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>> b/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>> index 899783f67580..aa546c972c3d 100644 >>>>>>>> --- a/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>> +++ b/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>> @@ -1637,6 +1637,39 @@ int vb2_core_prepare_buf(struct vb2_queue >>>>>>>> *q, >>>>>>>> unsigned int index, void *pb) >>>>>>>> } >>>>>>>> EXPORT_SYMBOL_GPL(vb2_core_prepare_buf); >>>>>>>> >>>>>>>> +int vb2_core_delete_buf(struct vb2_queue *q, unsigned int index) >>>>>>>> +{ >>>>>>>> + struct vb2_buffer *vb; >>>>>>>> + >>>>>>>> + vb = vb2_get_buffer(q, index); >>>>>>>> + if (!vb) { >>>>>>>> + dprintk(q, 1, "invalid buffer index %d\n", index); >>>>>>>> + return -EINVAL; >>>>>>>> + } >>>>>>>> + >>>>>>>> + if (vb->state != VB2_BUF_STATE_DEQUEUED) { >>>>>>>> + dprintk(q, 1, "can't delete non dequeued buffer >>>>>>>> index >>>>>>>> %d\n", index); >>>>>>>> + return -EINVAL; >>>>>>>> + } >>>>>>>> + >>>>>>> I know the driver could implement its own >>>>>>> v4l2_ioctl_ops->vidioc_delete_buf() that check whether a buffer is >>>>>>> used by the hardware as a future reference frame. >>>>>>> But I think we need a flag to let the user know which buffer is >>>>>>> still >>>>>>> used by the hardware. >>>>>>> Alternative ref case is safe, we only know it's existing when it is >>>>>>> dequeued in current V4L2 buffer mechanism. >>>>>>> While the Golden reference frame, such long term reference frame >>>>>>> could >>>>>>> last much longer. >>>>>> >>>>>> It is up to userland stack to know frames life time, it got the >>>>>> information for that. >>>>> >>>>> That is true for the stateless codec driver. >>>>> >>>>> While application for stateful decoder could never do that. It also >>>>> breaks what the document said: >>>>> >>>>> "The backing memory of |CAPTURE| buffers that are used as reference >>>>> frames by the stream may be read by the hardware even after they are >>>>> dequeued. Consequently, the client should avoid writing into this >>>>> memory >>>>> while the |CAPTURE| queue is streaming. Failure to observe this may >>>>> result in corruption of decoded frames." >>>>> >>>>>> >>>>>>>> + if (vb->planes[0].mem_priv) >>>>>>>> + call_void_vb_qop(vb, buf_cleanup, vb); >>>>>>>> + >>>>>>>> + /* Free MMAP buffers or release USERPTR buffers */ >>>>>>>> + if (q->memory == VB2_MEMORY_MMAP) >>>>>>>> + __vb2_buf_mem_free(vb); >>>> >>>> Here is another problem for the existing application, the mmap() from >>>> the mmap offset or exportbuffer fd would not create a reference to >>>> buffer in this step(while the exportbuffer would create one itself). >>>> >>>> When you delete a buffer, you may not release it from its virtual >>>> memory space, leaving a corrupted virtual memory space. Also this >>>> behavior is right, because mmap(2) says: >>>> >>>> "After the mmap() call has returned, the file descriptor, fd, can >>>> be closed immediately without invalidating the map‐ping." >>> >>> Existing applications do not call DELETE_BUF ioctl and when call it >>> they will be aware that the buffer is removed. >>> I have done it in GStreamer: >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__gitlab.freedesktop.org_benjamin.gaignard1_gstreamer_-2D_commit_fca0fbc934f4440693ce0ff6c8dc8a2e5f5f17d9&d=DwIDaQ&c=7dfBJ8cXbWjhc0BhImu8wVIoUFmBzj1s88r8EGyM0UY&r=P4xb2_7biqBxD4LGGPrSV6j-jf3C3xlR7PXU-mLTeZE&m=TGH9toTzGRfO5aBsfaMvGbcOw-28q6cPmpX6vScbHjpCtaLtb-RuvBvsJ0z9RvAB&s=Ufl1ccfRZf2EhnfCBvnQzRJV9CDhGxl5spe9WNECspU&e= >>> >>> >> >> I have read that. >> >> There is not a VP8 parser in Gstreamer, while a parser would not work >> when deal with the secure video(userspace can't access the data >> context at all). >> >> Besides, this adds extra work for the application for a stateful codec >> driver. The application need to parser the bitstream and track the dpb. >> >> I don't mind if you could fix the nonfiction mechanism for those >> non-display frame and internal reference state. >> >> That could be requirement for codec firmware that its driver could >> support this DELETE_BUF ioctl() feature. > > Sorry I don't see the link with my patches here... > I have work on non-secure VP9 on stateless codec. > DELETE_BUF ioctl is optional and the main goal is to offer a way to > applications > to save memory if they know when they could delete buffers without risk. I try to explain why I think this design in not "complete". One problem resolved, more problems would occur. For non-secure video, those applications have worked: - It would break what stateful means here, application need to acquire(parse) the information that driver should offer. Or it would break the decoding model. - Your Gstreamer sample code or this design won't work for AV1. For all the future possible secure video: - This feature could never be used from the current design. > >> >>> >>> Regards, >>> Benjamin >>> >>>> >>>>>>>> + else if (q->memory == VB2_MEMORY_DMABUF) >>>>>>>> + __vb2_buf_dmabuf_put(vb); >>>>>>>> + else >>>>>>>> + __vb2_buf_userptr_put(vb); >>>>>>>> + >>>>>>>> + vb2_queue_remove_buffer(q, vb); >>>>>>>> + kfree(vb); >>>>>>>> + >>>>>>>> + dprintk(q, 2, "buffer %d deleted\n", index); >>>>>>>> + return 0; >>>>>>>> +} >>>>>>>> + >>>>>>>> /* >>>>>>>> * vb2_start_streaming() - Attempt to start streaming. >>>>>>>> * @q: videobuf2 queue >>>>>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>>>> b/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>>>> index 724135d41f7f..cea666c17b41 100644 >>>>>>>> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>>>> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>>>> @@ -751,6 +751,12 @@ int vb2_prepare_buf(struct vb2_queue *q, >>>>>>>> struct >>>>>>>> media_device *mdev, >>>>>>>> } >>>>>>>> EXPORT_SYMBOL_GPL(vb2_prepare_buf); >>>>>>>> >>>>>>>> +int vb2_delete_buf(struct vb2_queue *q, struct v4l2_buffer *b) >>>>>>>> +{ >>>>>>>> + return vb2_core_delete_buf(q, b->index); >>>>>>>> +} >>>>>>>> +EXPORT_SYMBOL_GPL(vb2_delete_buf); >>>>>>>> + >>>>>>>> int vb2_create_bufs(struct vb2_queue *q, struct >>>>>>>> v4l2_create_buffers >>>>>>>> *create) >>>>>>>> { >>>>>>>> unsigned requested_planes = 1; >>>>>>>> diff --git a/drivers/media/v4l2-core/v4l2-dev.c >>>>>>>> b/drivers/media/v4l2-core/v4l2-dev.c >>>>>>>> index f81279492682..80ace2e1e932 100644 >>>>>>>> --- a/drivers/media/v4l2-core/v4l2-dev.c >>>>>>>> +++ b/drivers/media/v4l2-core/v4l2-dev.c >>>>>>>> @@ -720,6 +720,7 @@ static void determine_valid_ioctls(struct >>>>>>>> video_device *vdev) >>>>>>>> SET_VALID_IOCTL(ops, VIDIOC_PREPARE_BUF, >>>>>>>> vidioc_prepare_buf); >>>>>>>> SET_VALID_IOCTL(ops, VIDIOC_STREAMON, >>>>>>>> vidioc_streamon); >>>>>>>> SET_VALID_IOCTL(ops, VIDIOC_STREAMOFF, >>>>>>>> vidioc_streamoff); >>>>>>>> + SET_VALID_IOCTL(ops, VIDIOC_DELETE_BUF, >>>>>>>> vidioc_delete_buf); >>>>>>>> } >>>>>>>> >>>>>>>> if (is_vid || is_vbi || is_meta) { >>>>>>>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>>>> b/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>>>> index a858acea6547..1c737279d3ef 100644 >>>>>>>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>>>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>>>> @@ -2156,6 +2156,15 @@ static int v4l_prepare_buf(const struct >>>>>>>> v4l2_ioctl_ops *ops, >>>>>>>> return ret ? ret : ops->vidioc_prepare_buf(file, fh, b); >>>>>>>> } >>>>>>>> >>>>>>>> +static int v4l_delete_buf(const struct v4l2_ioctl_ops *ops, >>>>>>>> + struct file *file, void *fh, void *arg) >>>>>>>> +{ >>>>>>>> + struct v4l2_buffer *b = arg; >>>>>>>> + int ret = check_fmt(file, b->type); >>>>>>>> + >>>>>>>> + return ret ? ret : ops->vidioc_delete_buf(file, fh, b); >>>>>>>> +} >>>>>>>> + >>>>>>>> static int v4l_g_parm(const struct v4l2_ioctl_ops *ops, >>>>>>>> struct file *file, void *fh, void >>>>>>>> *arg) >>>>>>>> { >>>>>>>> @@ -2905,6 +2914,7 @@ static const struct v4l2_ioctl_info >>>>>>>> v4l2_ioctls[] = { >>>>>>>> IOCTL_INFO(VIDIOC_ENUM_FREQ_BANDS, v4l_enum_freq_bands, >>>>>>>> v4l_print_freq_band, 0), >>>>>>>> IOCTL_INFO(VIDIOC_DBG_G_CHIP_INFO, v4l_dbg_g_chip_info, >>>>>>>> v4l_print_dbg_chip_info, INFO_FL_CLEAR(v4l2_dbg_chip_info, >>>>>>>> match)), >>>>>>>> IOCTL_INFO(VIDIOC_QUERY_EXT_CTRL, v4l_query_ext_ctrl, >>>>>>>> v4l_print_query_ext_ctrl, INFO_FL_CTRL | >>>>>>>> INFO_FL_CLEAR(v4l2_query_ext_ctrl, id)), >>>>>>>> + IOCTL_INFO(VIDIOC_DELETE_BUF, v4l_delete_buf, >>>>>>>> v4l_print_buffer, INFO_FL_QUEUE), >>>>>>>> }; >>>>>>>> #define V4L2_IOCTLS ARRAY_SIZE(v4l2_ioctls) >>>>>>>> >>>>>>>> diff --git a/include/media/v4l2-ioctl.h >>>>>>>> b/include/media/v4l2-ioctl.h >>>>>>>> index edb733f21604..2f232ed884c7 100644 >>>>>>>> --- a/include/media/v4l2-ioctl.h >>>>>>>> +++ b/include/media/v4l2-ioctl.h >>>>>>>> @@ -163,6 +163,8 @@ struct v4l2_fh; >>>>>>>> * :ref:`VIDIOC_CREATE_BUFS <vidioc_create_bufs>` ioctl >>>>>>>> * @vidioc_prepare_buf: pointer to the function that implements >>>>>>>> * :ref:`VIDIOC_PREPARE_BUF <vidioc_prepare_buf>` ioctl >>>>>>>> + * @vidioc_delete_buf: pointer to the function that implements >>>>>>>> + * :ref:`VIDIOC_DELETE_BUF <vidioc_delete_buf>` ioctl >>>>>>>> * @vidioc_overlay: pointer to the function that implements >>>>>>>> * :ref:`VIDIOC_OVERLAY <vidioc_overlay>` ioctl >>>>>>>> * @vidioc_g_fbuf: pointer to the function that implements >>>>>>>> @@ -422,6 +424,8 @@ struct v4l2_ioctl_ops { >>>>>>>> struct v4l2_create_buffers *b); >>>>>>>> int (*vidioc_prepare_buf)(struct file *file, void *fh, >>>>>>>> struct v4l2_buffer *b); >>>>>>>> + int (*vidioc_delete_buf)(struct file *file, void *fh, >>>>>>>> + struct v4l2_buffer *b); >>>>>>>> >>>>>>>> int (*vidioc_overlay)(struct file *file, void *fh, >>>>>>>> unsigned >>>>>>>> int i); >>>>>>>> int (*vidioc_g_fbuf)(struct file *file, void *fh, >>>>>>>> diff --git a/include/media/videobuf2-core.h >>>>>>>> b/include/media/videobuf2-core.h >>>>>>>> index 080b783d608d..0f9e68f76b77 100644 >>>>>>>> --- a/include/media/videobuf2-core.h >>>>>>>> +++ b/include/media/videobuf2-core.h >>>>>>>> @@ -840,6 +840,15 @@ int vb2_core_create_bufs(struct vb2_queue *q, >>>>>>>> enum vb2_memory memory, >>>>>>>> */ >>>>>>>> int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int >>>>>>>> index, >>>>>>>> void *pb); >>>>>>>> >>>>>>>> +/** >>>>>>>> + * vb2_core_delete_buf() - >>>>>>>> + * @q: pointer to &struct vb2_queue with videobuf2 queue. >>>>>>>> + * @index: id number of the buffer. >>>>>>>> + * >>>>>>>> + * Return: returns zero on success; an error code otherwise. >>>>>>>> + */ >>>>>>>> +int vb2_core_delete_buf(struct vb2_queue *q, unsigned int index); >>>>>>>> + >>>>>>>> /** >>>>>>>> * vb2_core_qbuf() - Queue a buffer from userspace >>>>>>>> * >>>>>>>> diff --git a/include/media/videobuf2-v4l2.h >>>>>>>> b/include/media/videobuf2-v4l2.h >>>>>>>> index 88a7a565170e..3beeb4c735f0 100644 >>>>>>>> --- a/include/media/videobuf2-v4l2.h >>>>>>>> +++ b/include/media/videobuf2-v4l2.h >>>>>>>> @@ -114,6 +114,17 @@ int vb2_create_bufs(struct vb2_queue *q, >>>>>>>> struct >>>>>>>> v4l2_create_buffers *create); >>>>>>>> */ >>>>>>>> int vb2_prepare_buf(struct vb2_queue *q, struct media_device >>>>>>>> *mdev, >>>>>>>> struct v4l2_buffer *b); >>>>>>>> +/** >>>>>>>> + * vb2_delete_buf() - Delete the buffer from the queue >>>>>>>> + * >>>>>>>> + * @q: pointer to &struct vb2_queue with videobuf2 queue. >>>>>>>> + * @b: buffer structure passed from userspace to >>>>>>>> + * &v4l2_ioctl_ops->vidioc_delete_buf handler in driver >>>>>>>> + * >>>>>>>> + * The return values from this function are intended to be >>>>>>>> directly >>>>>>>> returned >>>>>>>> + * from &v4l2_ioctl_ops->vidioc_delete_buf handler in driver. >>>>>>>> + */ >>>>>>>> +int vb2_delete_buf(struct vb2_queue *q, struct v4l2_buffer *b); >>>>>>>> >>>>>>>> /** >>>>>>>> * vb2_qbuf() - Queue a buffer from userspace >>>>>>>> diff --git a/include/uapi/linux/videodev2.h >>>>>>>> b/include/uapi/linux/videodev2.h >>>>>>>> index aee75eb9e686..31bba1915642 100644 >>>>>>>> --- a/include/uapi/linux/videodev2.h >>>>>>>> +++ b/include/uapi/linux/videodev2.h >>>>>>>> @@ -2702,6 +2702,8 @@ struct v4l2_create_buffers { >>>>>>>> #define VIDIOC_DBG_G_CHIP_INFO _IOWR('V', 102, struct >>>>>>>> v4l2_dbg_chip_info) >>>>>>>> >>>>>>>> #define VIDIOC_QUERY_EXT_CTRL _IOWR('V', 103, struct >>>>>>>> v4l2_query_ext_ctrl) >>>>>>>> +#define VIDIOC_DELETE_BUF _IOWR('V', 104, struct >>>>>>>> v4l2_buffer) >>>>>>>> + >>>>>>>> >>>>>>>> /* Reminder: when adding new ioctls please add support for >>>>>>>> them to >>>>>>>> drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */ >>>>>>>> -- >>>>>>>> 2.39.2 >>>>>>>> >>>>> -- >>>>> Hsia-Jun(Randy) Li >>>>> -- Hsia-Jun(Randy) Li ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v3 10/11] media: v4l2: Add DELETE_BUF ioctl 2023-07-03 9:20 ` Hsia-Jun Li @ 2023-07-03 10:35 ` Benjamin Gaignard 2023-07-03 11:06 ` Hsia-Jun Li [not found] ` <8ca2f66e-8ff9-e885-274f-51417b581b78@synaptics.com> 0 siblings, 2 replies; 27+ messages in thread From: Benjamin Gaignard @ 2023-07-03 10:35 UTC (permalink / raw) To: Hsia-Jun Li Cc: linux-media, linux-kernel, p.zabel, ming.qian, hverkuil-cisco, gregkh, ezequiel, linux-arm-kernel, linux-mediatek, linux-arm-msm, linux-rockchip, nicolas.dufresne, linux-staging, kernel, mchehab, tfiga, m.szyprowski, ayaka Le 03/07/2023 à 11:20, Hsia-Jun Li a écrit : > > On 7/3/23 16:52, Benjamin Gaignard wrote: >> CAUTION: Email originated externally, do not click links or open >> attachments unless you recognize the sender and know the content is >> safe. >> >> >> Le 03/07/2023 à 10:19, Hsia-Jun Li a écrit : >>> >>> On 7/3/23 16:12, Benjamin Gaignard wrote: >>>> CAUTION: Email originated externally, do not click links or open >>>> attachments unless you recognize the sender and know the content is >>>> safe. >>>> >>>> >>>> Le 30/06/2023 à 11:43, Hsia-Jun Li a écrit : >>>>> >>>>> On 6/27/23 16:47, Hsia-Jun Li wrote: >>>>>> CAUTION: Email originated externally, do not click links or open >>>>>> attachments unless you recognize the sender and know the content is >>>>>> safe. >>>>>> >>>>>> >>>>>> On 6/27/23 16:43, Benjamin Gaignard wrote: >>>>>>> CAUTION: Email originated externally, do not click links or open >>>>>>> attachments unless you recognize the sender and know the content is >>>>>>> safe. >>>>>>> >>>>>>> >>>>>>> Le 27/06/2023 à 09:30, Hsia-Jun Li a écrit : >>>>>>>> >>>>>>>> On 6/22/23 21:13, Benjamin Gaignard wrote: >>>>>>>>> CAUTION: Email originated externally, do not click links or open >>>>>>>>> attachments unless you recognize the sender and know the >>>>>>>>> content is >>>>>>>>> safe. >>>>>>>>> >>>>>>>>> >>>>>>>>> VIDIOC_DELETE_BUF ioctl allows to delete a buffer from a queue. >>>>>>>>> >>>>>>>>> Signed-off-by: Benjamin Gaignard >>>>>>>>> <benjamin.gaignard@collabora.com> >>>>>>>>> --- >>>>>>>>> .../userspace-api/media/v4l/user-func.rst | 1 + >>>>>>>>> .../media/v4l/vidioc-delete-buf.rst | 51 >>>>>>>>> +++++++++++++++++++ >>>>>>>>> .../media/common/videobuf2/videobuf2-core.c | 33 ++++++++++++ >>>>>>>>> .../media/common/videobuf2/videobuf2-v4l2.c | 6 +++ >>>>>>>>> drivers/media/v4l2-core/v4l2-dev.c | 1 + >>>>>>>>> drivers/media/v4l2-core/v4l2-ioctl.c | 10 ++++ >>>>>>>>> include/media/v4l2-ioctl.h | 4 ++ >>>>>>>>> include/media/videobuf2-core.h | 9 ++++ >>>>>>>>> include/media/videobuf2-v4l2.h | 11 ++++ >>>>>>>>> include/uapi/linux/videodev2.h | 2 + >>>>>>>>> 10 files changed, 128 insertions(+) >>>>>>>>> create mode 100644 >>>>>>>>> Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>>>>> >>>>>>>>> diff --git a/Documentation/userspace-api/media/v4l/user-func.rst >>>>>>>>> b/Documentation/userspace-api/media/v4l/user-func.rst >>>>>>>>> index 15ff0bf7bbe6..8c74016e12fd 100644 >>>>>>>>> --- a/Documentation/userspace-api/media/v4l/user-func.rst >>>>>>>>> +++ b/Documentation/userspace-api/media/v4l/user-func.rst >>>>>>>>> @@ -17,6 +17,7 @@ Function Reference >>>>>>>>> vidioc-dbg-g-chip-info >>>>>>>>> vidioc-dbg-g-register >>>>>>>>> vidioc-decoder-cmd >>>>>>>>> + vidioc-delete-buf >>>>>>>>> vidioc-dqevent >>>>>>>>> vidioc-dv-timings-cap >>>>>>>>> vidioc-encoder-cmd >>>>>>>>> diff --git >>>>>>>>> a/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>>>>> b/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>>>>> new file mode 100644 >>>>>>>>> index 000000000000..0e7ce58f91bc >>>>>>>>> --- /dev/null >>>>>>>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>>>>> @@ -0,0 +1,51 @@ >>>>>>>>> +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later >>>>>>>>> +.. c:namespace:: V4L >>>>>>>>> + >>>>>>>>> +.. _VIDIOC_DELETE_BUF: >>>>>>>>> + >>>>>>>>> +************************ >>>>>>>>> +ioctl VIDIOC_DELETE_BUF >>>>>>>>> +************************ >>>>>>>>> + >>>>>>>>> +Name >>>>>>>>> +==== >>>>>>>>> + >>>>>>>>> +VIDIOC_DELETE_BUF - Delete a buffer from a queue >>>>>>>>> + >>>>>>>>> +Synopsis >>>>>>>>> +======== >>>>>>>>> + >>>>>>>>> +.. c:macro:: VIDIOC_DELETE_BUF >>>>>>>>> + >>>>>>>>> +``int ioctl(int fd, VIDIOC_DELETE_BUF, struct v4l2_buffer >>>>>>>>> *argp)`` >>>>>>>>> + >>>>>>>>> +Arguments >>>>>>>>> +========= >>>>>>>>> + >>>>>>>>> +``fd`` >>>>>>>>> + File descriptor returned by :c:func:`open()`. >>>>>>>>> + >>>>>>>>> +``argp`` >>>>>>>>> + Pointer to struct :c:type:`v4l2_buffer`. >>>>>>>>> + >>>>>>>>> +Description >>>>>>>>> +=========== >>>>>>>>> + >>>>>>>>> +Applications can optionally call the :ref:`VIDIOC_DELETE_BUF` >>>>>>>>> ioctl to >>>>>>>>> +delete a buffer from a queue. >>>>>>>>> + >>>>>>>>> +The struct :c:type:`v4l2_buffer` structure is specified in >>>>>>>>> +:ref:`buffer`. >>>>>>>>> + >>>>>>>>> +Return Value >>>>>>>>> +============ >>>>>>>>> + >>>>>>>>> +On success 0 is returned, on error -1 and the ``errno`` >>>>>>>>> variable is >>>>>>>>> set >>>>>>>>> +appropriately. The generic error codes are described at the >>>>>>>>> +:ref:`Generic Error Codes <gen-errors>` chapter. >>>>>>>>> + >>>>>>>>> +EBUSY >>>>>>>>> + File I/O is in progress. >>>>>>>>> + >>>>>>>>> +EINVAL >>>>>>>>> + The buffer ``index`` doesn't exist in the queue. >>>>>>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>>> b/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>>> index 899783f67580..aa546c972c3d 100644 >>>>>>>>> --- a/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>>> +++ b/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>>> @@ -1637,6 +1637,39 @@ int vb2_core_prepare_buf(struct vb2_queue >>>>>>>>> *q, >>>>>>>>> unsigned int index, void *pb) >>>>>>>>> } >>>>>>>>> EXPORT_SYMBOL_GPL(vb2_core_prepare_buf); >>>>>>>>> >>>>>>>>> +int vb2_core_delete_buf(struct vb2_queue *q, unsigned int index) >>>>>>>>> +{ >>>>>>>>> + struct vb2_buffer *vb; >>>>>>>>> + >>>>>>>>> + vb = vb2_get_buffer(q, index); >>>>>>>>> + if (!vb) { >>>>>>>>> + dprintk(q, 1, "invalid buffer index %d\n", >>>>>>>>> index); >>>>>>>>> + return -EINVAL; >>>>>>>>> + } >>>>>>>>> + >>>>>>>>> + if (vb->state != VB2_BUF_STATE_DEQUEUED) { >>>>>>>>> + dprintk(q, 1, "can't delete non dequeued buffer >>>>>>>>> index >>>>>>>>> %d\n", index); >>>>>>>>> + return -EINVAL; >>>>>>>>> + } >>>>>>>>> + >>>>>>>> I know the driver could implement its own >>>>>>>> v4l2_ioctl_ops->vidioc_delete_buf() that check whether a buffer is >>>>>>>> used by the hardware as a future reference frame. >>>>>>>> But I think we need a flag to let the user know which buffer is >>>>>>>> still >>>>>>>> used by the hardware. >>>>>>>> Alternative ref case is safe, we only know it's existing when >>>>>>>> it is >>>>>>>> dequeued in current V4L2 buffer mechanism. >>>>>>>> While the Golden reference frame, such long term reference frame >>>>>>>> could >>>>>>>> last much longer. >>>>>>> >>>>>>> It is up to userland stack to know frames life time, it got the >>>>>>> information for that. >>>>>> >>>>>> That is true for the stateless codec driver. >>>>>> >>>>>> While application for stateful decoder could never do that. It also >>>>>> breaks what the document said: >>>>>> >>>>>> "The backing memory of |CAPTURE| buffers that are used as reference >>>>>> frames by the stream may be read by the hardware even after they are >>>>>> dequeued. Consequently, the client should avoid writing into this >>>>>> memory >>>>>> while the |CAPTURE| queue is streaming. Failure to observe this may >>>>>> result in corruption of decoded frames." >>>>>> >>>>>>> >>>>>>>>> + if (vb->planes[0].mem_priv) >>>>>>>>> + call_void_vb_qop(vb, buf_cleanup, vb); >>>>>>>>> + >>>>>>>>> + /* Free MMAP buffers or release USERPTR buffers */ >>>>>>>>> + if (q->memory == VB2_MEMORY_MMAP) >>>>>>>>> + __vb2_buf_mem_free(vb); >>>>> >>>>> Here is another problem for the existing application, the mmap() from >>>>> the mmap offset or exportbuffer fd would not create a reference to >>>>> buffer in this step(while the exportbuffer would create one itself). >>>>> >>>>> When you delete a buffer, you may not release it from its virtual >>>>> memory space, leaving a corrupted virtual memory space. Also this >>>>> behavior is right, because mmap(2) says: >>>>> >>>>> "After the mmap() call has returned, the file descriptor, fd, can >>>>> be closed immediately without invalidating the map‐ping." >>>> >>>> Existing applications do not call DELETE_BUF ioctl and when call it >>>> they will be aware that the buffer is removed. >>>> I have done it in GStreamer: >>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__gitlab.freedesktop.org_benjamin.gaignard1_gstreamer_-2D_commit_fca0fbc934f4440693ce0ff6c8dc8a2e5f5f17d9&d=DwIDaQ&c=7dfBJ8cXbWjhc0BhImu8wVIoUFmBzj1s88r8EGyM0UY&r=P4xb2_7biqBxD4LGGPrSV6j-jf3C3xlR7PXU-mLTeZE&m=TGH9toTzGRfO5aBsfaMvGbcOw-28q6cPmpX6vScbHjpCtaLtb-RuvBvsJ0z9RvAB&s=Ufl1ccfRZf2EhnfCBvnQzRJV9CDhGxl5spe9WNECspU&e= >>>> >>>> >>> >>> I have read that. >>> >>> There is not a VP8 parser in Gstreamer, while a parser would not work >>> when deal with the secure video(userspace can't access the data >>> context at all). >>> >>> Besides, this adds extra work for the application for a stateful codec >>> driver. The application need to parser the bitstream and track the dpb. >>> >>> I don't mind if you could fix the nonfiction mechanism for those >>> non-display frame and internal reference state. >>> >>> That could be requirement for codec firmware that its driver could >>> support this DELETE_BUF ioctl() feature. >> >> Sorry I don't see the link with my patches here... >> I have work on non-secure VP9 on stateless codec. >> DELETE_BUF ioctl is optional and the main goal is to offer a way to >> applications >> to save memory if they know when they could delete buffers without risk. > > I try to explain why I think this design in not "complete". One > problem resolved, more problems would occur. > > > For non-secure video, those applications have worked: > > - It would break what stateful means here, application need to > acquire(parse) the information that driver should offer. > > Or it would break the decoding model. > > - Your Gstreamer sample code or this design won't work for AV1. why ? > > For all the future possible secure video: > > - This feature could never be used from the current design. > > >> >>> >>>> >>>> Regards, >>>> Benjamin >>>> >>>>> >>>>>>>>> + else if (q->memory == VB2_MEMORY_DMABUF) >>>>>>>>> + __vb2_buf_dmabuf_put(vb); >>>>>>>>> + else >>>>>>>>> + __vb2_buf_userptr_put(vb); >>>>>>>>> + >>>>>>>>> + vb2_queue_remove_buffer(q, vb); >>>>>>>>> + kfree(vb); >>>>>>>>> + >>>>>>>>> + dprintk(q, 2, "buffer %d deleted\n", index); >>>>>>>>> + return 0; >>>>>>>>> +} >>>>>>>>> + >>>>>>>>> /* >>>>>>>>> * vb2_start_streaming() - Attempt to start streaming. >>>>>>>>> * @q: videobuf2 queue >>>>>>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>>>>> b/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>>>>> index 724135d41f7f..cea666c17b41 100644 >>>>>>>>> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>>>>> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>>>>> @@ -751,6 +751,12 @@ int vb2_prepare_buf(struct vb2_queue *q, >>>>>>>>> struct >>>>>>>>> media_device *mdev, >>>>>>>>> } >>>>>>>>> EXPORT_SYMBOL_GPL(vb2_prepare_buf); >>>>>>>>> >>>>>>>>> +int vb2_delete_buf(struct vb2_queue *q, struct v4l2_buffer *b) >>>>>>>>> +{ >>>>>>>>> + return vb2_core_delete_buf(q, b->index); >>>>>>>>> +} >>>>>>>>> +EXPORT_SYMBOL_GPL(vb2_delete_buf); >>>>>>>>> + >>>>>>>>> int vb2_create_bufs(struct vb2_queue *q, struct >>>>>>>>> v4l2_create_buffers >>>>>>>>> *create) >>>>>>>>> { >>>>>>>>> unsigned requested_planes = 1; >>>>>>>>> diff --git a/drivers/media/v4l2-core/v4l2-dev.c >>>>>>>>> b/drivers/media/v4l2-core/v4l2-dev.c >>>>>>>>> index f81279492682..80ace2e1e932 100644 >>>>>>>>> --- a/drivers/media/v4l2-core/v4l2-dev.c >>>>>>>>> +++ b/drivers/media/v4l2-core/v4l2-dev.c >>>>>>>>> @@ -720,6 +720,7 @@ static void determine_valid_ioctls(struct >>>>>>>>> video_device *vdev) >>>>>>>>> SET_VALID_IOCTL(ops, VIDIOC_PREPARE_BUF, >>>>>>>>> vidioc_prepare_buf); >>>>>>>>> SET_VALID_IOCTL(ops, VIDIOC_STREAMON, >>>>>>>>> vidioc_streamon); >>>>>>>>> SET_VALID_IOCTL(ops, VIDIOC_STREAMOFF, >>>>>>>>> vidioc_streamoff); >>>>>>>>> + SET_VALID_IOCTL(ops, VIDIOC_DELETE_BUF, >>>>>>>>> vidioc_delete_buf); >>>>>>>>> } >>>>>>>>> >>>>>>>>> if (is_vid || is_vbi || is_meta) { >>>>>>>>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>>>>> b/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>>>>> index a858acea6547..1c737279d3ef 100644 >>>>>>>>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>>>>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>>>>> @@ -2156,6 +2156,15 @@ static int v4l_prepare_buf(const struct >>>>>>>>> v4l2_ioctl_ops *ops, >>>>>>>>> return ret ? ret : ops->vidioc_prepare_buf(file, fh, b); >>>>>>>>> } >>>>>>>>> >>>>>>>>> +static int v4l_delete_buf(const struct v4l2_ioctl_ops *ops, >>>>>>>>> + struct file *file, void *fh, void *arg) >>>>>>>>> +{ >>>>>>>>> + struct v4l2_buffer *b = arg; >>>>>>>>> + int ret = check_fmt(file, b->type); >>>>>>>>> + >>>>>>>>> + return ret ? ret : ops->vidioc_delete_buf(file, fh, b); >>>>>>>>> +} >>>>>>>>> + >>>>>>>>> static int v4l_g_parm(const struct v4l2_ioctl_ops *ops, >>>>>>>>> struct file *file, void *fh, >>>>>>>>> void >>>>>>>>> *arg) >>>>>>>>> { >>>>>>>>> @@ -2905,6 +2914,7 @@ static const struct v4l2_ioctl_info >>>>>>>>> v4l2_ioctls[] = { >>>>>>>>> IOCTL_INFO(VIDIOC_ENUM_FREQ_BANDS, v4l_enum_freq_bands, >>>>>>>>> v4l_print_freq_band, 0), >>>>>>>>> IOCTL_INFO(VIDIOC_DBG_G_CHIP_INFO, v4l_dbg_g_chip_info, >>>>>>>>> v4l_print_dbg_chip_info, INFO_FL_CLEAR(v4l2_dbg_chip_info, >>>>>>>>> match)), >>>>>>>>> IOCTL_INFO(VIDIOC_QUERY_EXT_CTRL, v4l_query_ext_ctrl, >>>>>>>>> v4l_print_query_ext_ctrl, INFO_FL_CTRL | >>>>>>>>> INFO_FL_CLEAR(v4l2_query_ext_ctrl, id)), >>>>>>>>> + IOCTL_INFO(VIDIOC_DELETE_BUF, v4l_delete_buf, >>>>>>>>> v4l_print_buffer, INFO_FL_QUEUE), >>>>>>>>> }; >>>>>>>>> #define V4L2_IOCTLS ARRAY_SIZE(v4l2_ioctls) >>>>>>>>> >>>>>>>>> diff --git a/include/media/v4l2-ioctl.h >>>>>>>>> b/include/media/v4l2-ioctl.h >>>>>>>>> index edb733f21604..2f232ed884c7 100644 >>>>>>>>> --- a/include/media/v4l2-ioctl.h >>>>>>>>> +++ b/include/media/v4l2-ioctl.h >>>>>>>>> @@ -163,6 +163,8 @@ struct v4l2_fh; >>>>>>>>> * :ref:`VIDIOC_CREATE_BUFS <vidioc_create_bufs>` ioctl >>>>>>>>> * @vidioc_prepare_buf: pointer to the function that implements >>>>>>>>> * :ref:`VIDIOC_PREPARE_BUF <vidioc_prepare_buf>` ioctl >>>>>>>>> + * @vidioc_delete_buf: pointer to the function that implements >>>>>>>>> + * :ref:`VIDIOC_DELETE_BUF <vidioc_delete_buf>` ioctl >>>>>>>>> * @vidioc_overlay: pointer to the function that implements >>>>>>>>> * :ref:`VIDIOC_OVERLAY <vidioc_overlay>` ioctl >>>>>>>>> * @vidioc_g_fbuf: pointer to the function that implements >>>>>>>>> @@ -422,6 +424,8 @@ struct v4l2_ioctl_ops { >>>>>>>>> struct v4l2_create_buffers >>>>>>>>> *b); >>>>>>>>> int (*vidioc_prepare_buf)(struct file *file, void *fh, >>>>>>>>> struct v4l2_buffer *b); >>>>>>>>> + int (*vidioc_delete_buf)(struct file *file, void *fh, >>>>>>>>> + struct v4l2_buffer *b); >>>>>>>>> >>>>>>>>> int (*vidioc_overlay)(struct file *file, void *fh, >>>>>>>>> unsigned >>>>>>>>> int i); >>>>>>>>> int (*vidioc_g_fbuf)(struct file *file, void *fh, >>>>>>>>> diff --git a/include/media/videobuf2-core.h >>>>>>>>> b/include/media/videobuf2-core.h >>>>>>>>> index 080b783d608d..0f9e68f76b77 100644 >>>>>>>>> --- a/include/media/videobuf2-core.h >>>>>>>>> +++ b/include/media/videobuf2-core.h >>>>>>>>> @@ -840,6 +840,15 @@ int vb2_core_create_bufs(struct vb2_queue >>>>>>>>> *q, >>>>>>>>> enum vb2_memory memory, >>>>>>>>> */ >>>>>>>>> int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int >>>>>>>>> index, >>>>>>>>> void *pb); >>>>>>>>> >>>>>>>>> +/** >>>>>>>>> + * vb2_core_delete_buf() - >>>>>>>>> + * @q: pointer to &struct vb2_queue with videobuf2 queue. >>>>>>>>> + * @index: id number of the buffer. >>>>>>>>> + * >>>>>>>>> + * Return: returns zero on success; an error code otherwise. >>>>>>>>> + */ >>>>>>>>> +int vb2_core_delete_buf(struct vb2_queue *q, unsigned int >>>>>>>>> index); >>>>>>>>> + >>>>>>>>> /** >>>>>>>>> * vb2_core_qbuf() - Queue a buffer from userspace >>>>>>>>> * >>>>>>>>> diff --git a/include/media/videobuf2-v4l2.h >>>>>>>>> b/include/media/videobuf2-v4l2.h >>>>>>>>> index 88a7a565170e..3beeb4c735f0 100644 >>>>>>>>> --- a/include/media/videobuf2-v4l2.h >>>>>>>>> +++ b/include/media/videobuf2-v4l2.h >>>>>>>>> @@ -114,6 +114,17 @@ int vb2_create_bufs(struct vb2_queue *q, >>>>>>>>> struct >>>>>>>>> v4l2_create_buffers *create); >>>>>>>>> */ >>>>>>>>> int vb2_prepare_buf(struct vb2_queue *q, struct media_device >>>>>>>>> *mdev, >>>>>>>>> struct v4l2_buffer *b); >>>>>>>>> +/** >>>>>>>>> + * vb2_delete_buf() - Delete the buffer from the queue >>>>>>>>> + * >>>>>>>>> + * @q: pointer to &struct vb2_queue with videobuf2 >>>>>>>>> queue. >>>>>>>>> + * @b: buffer structure passed from userspace to >>>>>>>>> + * &v4l2_ioctl_ops->vidioc_delete_buf handler in driver >>>>>>>>> + * >>>>>>>>> + * The return values from this function are intended to be >>>>>>>>> directly >>>>>>>>> returned >>>>>>>>> + * from &v4l2_ioctl_ops->vidioc_delete_buf handler in driver. >>>>>>>>> + */ >>>>>>>>> +int vb2_delete_buf(struct vb2_queue *q, struct v4l2_buffer *b); >>>>>>>>> >>>>>>>>> /** >>>>>>>>> * vb2_qbuf() - Queue a buffer from userspace >>>>>>>>> diff --git a/include/uapi/linux/videodev2.h >>>>>>>>> b/include/uapi/linux/videodev2.h >>>>>>>>> index aee75eb9e686..31bba1915642 100644 >>>>>>>>> --- a/include/uapi/linux/videodev2.h >>>>>>>>> +++ b/include/uapi/linux/videodev2.h >>>>>>>>> @@ -2702,6 +2702,8 @@ struct v4l2_create_buffers { >>>>>>>>> #define VIDIOC_DBG_G_CHIP_INFO _IOWR('V', 102, struct >>>>>>>>> v4l2_dbg_chip_info) >>>>>>>>> >>>>>>>>> #define VIDIOC_QUERY_EXT_CTRL _IOWR('V', 103, struct >>>>>>>>> v4l2_query_ext_ctrl) >>>>>>>>> +#define VIDIOC_DELETE_BUF _IOWR('V', 104, struct >>>>>>>>> v4l2_buffer) >>>>>>>>> + >>>>>>>>> >>>>>>>>> /* Reminder: when adding new ioctls please add support for >>>>>>>>> them to >>>>>>>>> drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */ >>>>>>>>> -- >>>>>>>>> 2.39.2 >>>>>>>>> >>>>>> -- >>>>>> Hsia-Jun(Randy) Li >>>>>> ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v3 10/11] media: v4l2: Add DELETE_BUF ioctl 2023-07-03 10:35 ` Benjamin Gaignard @ 2023-07-03 11:06 ` Hsia-Jun Li [not found] ` <8ca2f66e-8ff9-e885-274f-51417b581b78@synaptics.com> 1 sibling, 0 replies; 27+ messages in thread From: Hsia-Jun Li @ 2023-07-03 11:06 UTC (permalink / raw) To: Benjamin Gaignard Cc: linux-media, linux-kernel, p.zabel, ming.qian, hverkuil-cisco, gregkh, ezequiel, linux-arm-kernel, linux-mediatek, linux-arm-msm, linux-rockchip, nicolas.dufresne, linux-staging, kernel, mchehab, tfiga, m.szyprowski, ayaka Sorry for html version, again. On 7/3/23 18:35, Benjamin Gaignard wrote: > CAUTION: Email originated externally, do not click links or open > attachments unless you recognize the sender and know the content is safe. > > > Le 03/07/2023 à 11:20, Hsia-Jun Li a écrit : >> >> On 7/3/23 16:52, Benjamin Gaignard wrote: >>> CAUTION: Email originated externally, do not click links or open >>> attachments unless you recognize the sender and know the content is >>> safe. >>> >>> >>> Le 03/07/2023 à 10:19, Hsia-Jun Li a écrit : >>>> >>>> On 7/3/23 16:12, Benjamin Gaignard wrote: >>>>> CAUTION: Email originated externally, do not click links or open >>>>> attachments unless you recognize the sender and know the content is >>>>> safe. >>>>> >>>>> >>>>> Le 30/06/2023 à 11:43, Hsia-Jun Li a écrit : >>>>>> >>>>>> On 6/27/23 16:47, Hsia-Jun Li wrote: >>>>>>> CAUTION: Email originated externally, do not click links or open >>>>>>> attachments unless you recognize the sender and know the content is >>>>>>> safe. >>>>>>> >>>>>>> >>>>>>> On 6/27/23 16:43, Benjamin Gaignard wrote: >>>>>>>> CAUTION: Email originated externally, do not click links or open >>>>>>>> attachments unless you recognize the sender and know the >>>>>>>> content is >>>>>>>> safe. >>>>>>>> >>>>>>>> >>>>>>>> Le 27/06/2023 à 09:30, Hsia-Jun Li a écrit : >>>>>>>>> >>>>>>>>> On 6/22/23 21:13, Benjamin Gaignard wrote: >>>>>>>>>> CAUTION: Email originated externally, do not click links or open >>>>>>>>>> attachments unless you recognize the sender and know the >>>>>>>>>> content is >>>>>>>>>> safe. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> VIDIOC_DELETE_BUF ioctl allows to delete a buffer from a queue. >>>>>>>>>> >>>>>>>>>> Signed-off-by: Benjamin Gaignard >>>>>>>>>> <benjamin.gaignard@collabora.com> >>>>>>>>>> --- >>>>>>>>>> .../userspace-api/media/v4l/user-func.rst | 1 + >>>>>>>>>> .../media/v4l/vidioc-delete-buf.rst | 51 >>>>>>>>>> +++++++++++++++++++ >>>>>>>>>> .../media/common/videobuf2/videobuf2-core.c | 33 ++++++++++++ >>>>>>>>>> .../media/common/videobuf2/videobuf2-v4l2.c | 6 +++ >>>>>>>>>> drivers/media/v4l2-core/v4l2-dev.c | 1 + >>>>>>>>>> drivers/media/v4l2-core/v4l2-ioctl.c | 10 ++++ >>>>>>>>>> include/media/v4l2-ioctl.h | 4 ++ >>>>>>>>>> include/media/videobuf2-core.h | 9 ++++ >>>>>>>>>> include/media/videobuf2-v4l2.h | 11 ++++ >>>>>>>>>> include/uapi/linux/videodev2.h | 2 + >>>>>>>>>> 10 files changed, 128 insertions(+) >>>>>>>>>> create mode 100644 >>>>>>>>>> Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>>>>>> >>>>>>>>>> diff --git a/Documentation/userspace-api/media/v4l/user-func.rst >>>>>>>>>> b/Documentation/userspace-api/media/v4l/user-func.rst >>>>>>>>>> index 15ff0bf7bbe6..8c74016e12fd 100644 >>>>>>>>>> --- a/Documentation/userspace-api/media/v4l/user-func.rst >>>>>>>>>> +++ b/Documentation/userspace-api/media/v4l/user-func.rst >>>>>>>>>> @@ -17,6 +17,7 @@ Function Reference >>>>>>>>>> vidioc-dbg-g-chip-info >>>>>>>>>> vidioc-dbg-g-register >>>>>>>>>> vidioc-decoder-cmd >>>>>>>>>> + vidioc-delete-buf >>>>>>>>>> vidioc-dqevent >>>>>>>>>> vidioc-dv-timings-cap >>>>>>>>>> vidioc-encoder-cmd >>>>>>>>>> diff --git >>>>>>>>>> a/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>>>>>> b/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>>>>>> new file mode 100644 >>>>>>>>>> index 000000000000..0e7ce58f91bc >>>>>>>>>> --- /dev/null >>>>>>>>>> +++ >>>>>>>>>> b/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>>>>>> @@ -0,0 +1,51 @@ >>>>>>>>>> +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later >>>>>>>>>> +.. c:namespace:: V4L >>>>>>>>>> + >>>>>>>>>> +.. _VIDIOC_DELETE_BUF: >>>>>>>>>> + >>>>>>>>>> +************************ >>>>>>>>>> +ioctl VIDIOC_DELETE_BUF >>>>>>>>>> +************************ >>>>>>>>>> + >>>>>>>>>> +Name >>>>>>>>>> +==== >>>>>>>>>> + >>>>>>>>>> +VIDIOC_DELETE_BUF - Delete a buffer from a queue >>>>>>>>>> + >>>>>>>>>> +Synopsis >>>>>>>>>> +======== >>>>>>>>>> + >>>>>>>>>> +.. c:macro:: VIDIOC_DELETE_BUF >>>>>>>>>> + >>>>>>>>>> +``int ioctl(int fd, VIDIOC_DELETE_BUF, struct v4l2_buffer >>>>>>>>>> *argp)`` >>>>>>>>>> + >>>>>>>>>> +Arguments >>>>>>>>>> +========= >>>>>>>>>> + >>>>>>>>>> +``fd`` >>>>>>>>>> + File descriptor returned by :c:func:`open()`. >>>>>>>>>> + >>>>>>>>>> +``argp`` >>>>>>>>>> + Pointer to struct :c:type:`v4l2_buffer`. >>>>>>>>>> + >>>>>>>>>> +Description >>>>>>>>>> +=========== >>>>>>>>>> + >>>>>>>>>> +Applications can optionally call the :ref:`VIDIOC_DELETE_BUF` >>>>>>>>>> ioctl to >>>>>>>>>> +delete a buffer from a queue. >>>>>>>>>> + >>>>>>>>>> +The struct :c:type:`v4l2_buffer` structure is specified in >>>>>>>>>> +:ref:`buffer`. >>>>>>>>>> + >>>>>>>>>> +Return Value >>>>>>>>>> +============ >>>>>>>>>> + >>>>>>>>>> +On success 0 is returned, on error -1 and the ``errno`` >>>>>>>>>> variable is >>>>>>>>>> set >>>>>>>>>> +appropriately. The generic error codes are described at the >>>>>>>>>> +:ref:`Generic Error Codes <gen-errors>` chapter. >>>>>>>>>> + >>>>>>>>>> +EBUSY >>>>>>>>>> + File I/O is in progress. >>>>>>>>>> + >>>>>>>>>> +EINVAL >>>>>>>>>> + The buffer ``index`` doesn't exist in the queue. >>>>>>>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>>>> b/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>>>> index 899783f67580..aa546c972c3d 100644 >>>>>>>>>> --- a/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>>>> +++ b/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>>>> @@ -1637,6 +1637,39 @@ int vb2_core_prepare_buf(struct vb2_queue >>>>>>>>>> *q, >>>>>>>>>> unsigned int index, void *pb) >>>>>>>>>> } >>>>>>>>>> EXPORT_SYMBOL_GPL(vb2_core_prepare_buf); >>>>>>>>>> >>>>>>>>>> +int vb2_core_delete_buf(struct vb2_queue *q, unsigned int >>>>>>>>>> index) >>>>>>>>>> +{ >>>>>>>>>> + struct vb2_buffer *vb; >>>>>>>>>> + >>>>>>>>>> + vb = vb2_get_buffer(q, index); >>>>>>>>>> + if (!vb) { >>>>>>>>>> + dprintk(q, 1, "invalid buffer index %d\n", >>>>>>>>>> index); >>>>>>>>>> + return -EINVAL; >>>>>>>>>> + } >>>>>>>>>> + >>>>>>>>>> + if (vb->state != VB2_BUF_STATE_DEQUEUED) { >>>>>>>>>> + dprintk(q, 1, "can't delete non dequeued buffer >>>>>>>>>> index >>>>>>>>>> %d\n", index); >>>>>>>>>> + return -EINVAL; >>>>>>>>>> + } >>>>>>>>>> + >>>>>>>>> I know the driver could implement its own >>>>>>>>> v4l2_ioctl_ops->vidioc_delete_buf() that check whether a >>>>>>>>> buffer is >>>>>>>>> used by the hardware as a future reference frame. >>>>>>>>> But I think we need a flag to let the user know which buffer is >>>>>>>>> still >>>>>>>>> used by the hardware. >>>>>>>>> Alternative ref case is safe, we only know it's existing when >>>>>>>>> it is >>>>>>>>> dequeued in current V4L2 buffer mechanism. >>>>>>>>> While the Golden reference frame, such long term reference frame >>>>>>>>> could >>>>>>>>> last much longer. >>>>>>>> >>>>>>>> It is up to userland stack to know frames life time, it got the >>>>>>>> information for that. >>>>>>> >>>>>>> That is true for the stateless codec driver. >>>>>>> >>>>>>> While application for stateful decoder could never do that. It also >>>>>>> breaks what the document said: >>>>>>> >>>>>>> "The backing memory of |CAPTURE| buffers that are used as reference >>>>>>> frames by the stream may be read by the hardware even after they >>>>>>> are >>>>>>> dequeued. Consequently, the client should avoid writing into this >>>>>>> memory >>>>>>> while the |CAPTURE| queue is streaming. Failure to observe this may >>>>>>> result in corruption of decoded frames." >>>>>>> >>>>>>>> >>>>>>>>>> + if (vb->planes[0].mem_priv) >>>>>>>>>> + call_void_vb_qop(vb, buf_cleanup, vb); >>>>>>>>>> + >>>>>>>>>> + /* Free MMAP buffers or release USERPTR buffers */ >>>>>>>>>> + if (q->memory == VB2_MEMORY_MMAP) >>>>>>>>>> + __vb2_buf_mem_free(vb); >>>>>> >>>>>> Here is another problem for the existing application, the mmap() >>>>>> from >>>>>> the mmap offset or exportbuffer fd would not create a reference to >>>>>> buffer in this step(while the exportbuffer would create one itself). >>>>>> >>>>>> When you delete a buffer, you may not release it from its virtual >>>>>> memory space, leaving a corrupted virtual memory space. Also this >>>>>> behavior is right, because mmap(2) says: >>>>>> >>>>>> "After the mmap() call has returned, the file descriptor, fd, can >>>>>> be closed immediately without invalidating the map‐ping." >>>>> >>>>> Existing applications do not call DELETE_BUF ioctl and when call it >>>>> they will be aware that the buffer is removed. >>>>> I have done it in GStreamer: >>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__gitlab.freedesktop.org_benjamin.gaignard1_gstreamer_-2D_commit_fca0fbc934f4440693ce0ff6c8dc8a2e5f5f17d9&d=DwIDaQ&c=7dfBJ8cXbWjhc0BhImu8wVIoUFmBzj1s88r8EGyM0UY&r=P4xb2_7biqBxD4LGGPrSV6j-jf3C3xlR7PXU-mLTeZE&m=TGH9toTzGRfO5aBsfaMvGbcOw-28q6cPmpX6vScbHjpCtaLtb-RuvBvsJ0z9RvAB&s=Ufl1ccfRZf2EhnfCBvnQzRJV9CDhGxl5spe9WNECspU&e= >>>>> >>>>> >>>>> >>>> >>>> I have read that. >>>> >>>> There is not a VP8 parser in Gstreamer, while a parser would not work >>>> when deal with the secure video(userspace can't access the data >>>> context at all). >>>> >>>> Besides, this adds extra work for the application for a stateful codec >>>> driver. The application need to parser the bitstream and track the >>>> dpb. >>>> >>>> I don't mind if you could fix the nonfiction mechanism for those >>>> non-display frame and internal reference state. >>>> >>>> That could be requirement for codec firmware that its driver could >>>> support this DELETE_BUF ioctl() feature. >>> >>> Sorry I don't see the link with my patches here... >>> I have work on non-secure VP9 on stateless codec. >>> DELETE_BUF ioctl is optional and the main goal is to offer a way to >>> applications >>> to save memory if they know when they could delete buffers without >>> risk. >> >> I try to explain why I think this design in not "complete". One >> problem resolved, more problems would occur. >> >> >> For non-secure video, those applications have worked: >> >> - It would break what stateful means here, application need to >> acquire(parse) the information that driver should offer. >> >> Or it would break the decoding model. >> >> - Your Gstreamer sample code or this design won't work for AV1. > why ? A frame in AV1 could be put to display order with a future frame. That is quite different to those ITU codecs. Supposing a frame (we call it frame 0) without show_display frame, you could delete it (just want to do that) once it is decoded. Actually, unless 7 future frames have came and parser has acknowledged them, it is not safe to delete that frame. Or a show_exist that make frame 0 present. In my personal opinion, this feature could break many uAPI's, I have said my worrying in the other emails. >> >> For all the future possible secure video: >> >> - This feature could never be used from the current design. >> >> >>> >>>> >>>>> >>>>> Regards, >>>>> Benjamin >>>>> >>>>>> >>>>>>>>>> + else if (q->memory == VB2_MEMORY_DMABUF) >>>>>>>>>> + __vb2_buf_dmabuf_put(vb); >>>>>>>>>> + else >>>>>>>>>> + __vb2_buf_userptr_put(vb); >>>>>>>>>> + >>>>>>>>>> + vb2_queue_remove_buffer(q, vb); >>>>>>>>>> + kfree(vb); >>>>>>>>>> + >>>>>>>>>> + dprintk(q, 2, "buffer %d deleted\n", index); >>>>>>>>>> + return 0; >>>>>>>>>> +} >>>>>>>>>> + >>>>>>>>>> /* >>>>>>>>>> * vb2_start_streaming() - Attempt to start streaming. >>>>>>>>>> * @q: videobuf2 queue >>>>>>>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>>>>>> b/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>>>>>> index 724135d41f7f..cea666c17b41 100644 >>>>>>>>>> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>>>>>> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>>>>>> @@ -751,6 +751,12 @@ int vb2_prepare_buf(struct vb2_queue *q, >>>>>>>>>> struct >>>>>>>>>> media_device *mdev, >>>>>>>>>> } >>>>>>>>>> EXPORT_SYMBOL_GPL(vb2_prepare_buf); >>>>>>>>>> >>>>>>>>>> +int vb2_delete_buf(struct vb2_queue *q, struct v4l2_buffer *b) >>>>>>>>>> +{ >>>>>>>>>> + return vb2_core_delete_buf(q, b->index); >>>>>>>>>> +} >>>>>>>>>> +EXPORT_SYMBOL_GPL(vb2_delete_buf); >>>>>>>>>> + >>>>>>>>>> int vb2_create_bufs(struct vb2_queue *q, struct >>>>>>>>>> v4l2_create_buffers >>>>>>>>>> *create) >>>>>>>>>> { >>>>>>>>>> unsigned requested_planes = 1; >>>>>>>>>> diff --git a/drivers/media/v4l2-core/v4l2-dev.c >>>>>>>>>> b/drivers/media/v4l2-core/v4l2-dev.c >>>>>>>>>> index f81279492682..80ace2e1e932 100644 >>>>>>>>>> --- a/drivers/media/v4l2-core/v4l2-dev.c >>>>>>>>>> +++ b/drivers/media/v4l2-core/v4l2-dev.c >>>>>>>>>> @@ -720,6 +720,7 @@ static void determine_valid_ioctls(struct >>>>>>>>>> video_device *vdev) >>>>>>>>>> SET_VALID_IOCTL(ops, VIDIOC_PREPARE_BUF, >>>>>>>>>> vidioc_prepare_buf); >>>>>>>>>> SET_VALID_IOCTL(ops, VIDIOC_STREAMON, >>>>>>>>>> vidioc_streamon); >>>>>>>>>> SET_VALID_IOCTL(ops, VIDIOC_STREAMOFF, >>>>>>>>>> vidioc_streamoff); >>>>>>>>>> + SET_VALID_IOCTL(ops, VIDIOC_DELETE_BUF, >>>>>>>>>> vidioc_delete_buf); >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> if (is_vid || is_vbi || is_meta) { >>>>>>>>>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>>>>>> b/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>>>>>> index a858acea6547..1c737279d3ef 100644 >>>>>>>>>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>>>>>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>>>>>> @@ -2156,6 +2156,15 @@ static int v4l_prepare_buf(const struct >>>>>>>>>> v4l2_ioctl_ops *ops, >>>>>>>>>> return ret ? ret : ops->vidioc_prepare_buf(file, fh, >>>>>>>>>> b); >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> +static int v4l_delete_buf(const struct v4l2_ioctl_ops *ops, >>>>>>>>>> + struct file *file, void *fh, void >>>>>>>>>> *arg) >>>>>>>>>> +{ >>>>>>>>>> + struct v4l2_buffer *b = arg; >>>>>>>>>> + int ret = check_fmt(file, b->type); >>>>>>>>>> + >>>>>>>>>> + return ret ? ret : ops->vidioc_delete_buf(file, fh, b); >>>>>>>>>> +} >>>>>>>>>> + >>>>>>>>>> static int v4l_g_parm(const struct v4l2_ioctl_ops *ops, >>>>>>>>>> struct file *file, void *fh, >>>>>>>>>> void >>>>>>>>>> *arg) >>>>>>>>>> { >>>>>>>>>> @@ -2905,6 +2914,7 @@ static const struct v4l2_ioctl_info >>>>>>>>>> v4l2_ioctls[] = { >>>>>>>>>> IOCTL_INFO(VIDIOC_ENUM_FREQ_BANDS, v4l_enum_freq_bands, >>>>>>>>>> v4l_print_freq_band, 0), >>>>>>>>>> IOCTL_INFO(VIDIOC_DBG_G_CHIP_INFO, v4l_dbg_g_chip_info, >>>>>>>>>> v4l_print_dbg_chip_info, INFO_FL_CLEAR(v4l2_dbg_chip_info, >>>>>>>>>> match)), >>>>>>>>>> IOCTL_INFO(VIDIOC_QUERY_EXT_CTRL, v4l_query_ext_ctrl, >>>>>>>>>> v4l_print_query_ext_ctrl, INFO_FL_CTRL | >>>>>>>>>> INFO_FL_CLEAR(v4l2_query_ext_ctrl, id)), >>>>>>>>>> + IOCTL_INFO(VIDIOC_DELETE_BUF, v4l_delete_buf, >>>>>>>>>> v4l_print_buffer, INFO_FL_QUEUE), >>>>>>>>>> }; >>>>>>>>>> #define V4L2_IOCTLS ARRAY_SIZE(v4l2_ioctls) >>>>>>>>>> >>>>>>>>>> diff --git a/include/media/v4l2-ioctl.h >>>>>>>>>> b/include/media/v4l2-ioctl.h >>>>>>>>>> index edb733f21604..2f232ed884c7 100644 >>>>>>>>>> --- a/include/media/v4l2-ioctl.h >>>>>>>>>> +++ b/include/media/v4l2-ioctl.h >>>>>>>>>> @@ -163,6 +163,8 @@ struct v4l2_fh; >>>>>>>>>> * :ref:`VIDIOC_CREATE_BUFS <vidioc_create_bufs>` ioctl >>>>>>>>>> * @vidioc_prepare_buf: pointer to the function that >>>>>>>>>> implements >>>>>>>>>> * :ref:`VIDIOC_PREPARE_BUF <vidioc_prepare_buf>` ioctl >>>>>>>>>> + * @vidioc_delete_buf: pointer to the function that implements >>>>>>>>>> + * :ref:`VIDIOC_DELETE_BUF <vidioc_delete_buf>` ioctl >>>>>>>>>> * @vidioc_overlay: pointer to the function that implements >>>>>>>>>> * :ref:`VIDIOC_OVERLAY <vidioc_overlay>` ioctl >>>>>>>>>> * @vidioc_g_fbuf: pointer to the function that implements >>>>>>>>>> @@ -422,6 +424,8 @@ struct v4l2_ioctl_ops { >>>>>>>>>> struct v4l2_create_buffers >>>>>>>>>> *b); >>>>>>>>>> int (*vidioc_prepare_buf)(struct file *file, void *fh, >>>>>>>>>> struct v4l2_buffer *b); >>>>>>>>>> + int (*vidioc_delete_buf)(struct file *file, void *fh, >>>>>>>>>> + struct v4l2_buffer *b); >>>>>>>>>> >>>>>>>>>> int (*vidioc_overlay)(struct file *file, void *fh, >>>>>>>>>> unsigned >>>>>>>>>> int i); >>>>>>>>>> int (*vidioc_g_fbuf)(struct file *file, void *fh, >>>>>>>>>> diff --git a/include/media/videobuf2-core.h >>>>>>>>>> b/include/media/videobuf2-core.h >>>>>>>>>> index 080b783d608d..0f9e68f76b77 100644 >>>>>>>>>> --- a/include/media/videobuf2-core.h >>>>>>>>>> +++ b/include/media/videobuf2-core.h >>>>>>>>>> @@ -840,6 +840,15 @@ int vb2_core_create_bufs(struct vb2_queue >>>>>>>>>> *q, >>>>>>>>>> enum vb2_memory memory, >>>>>>>>>> */ >>>>>>>>>> int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int >>>>>>>>>> index, >>>>>>>>>> void *pb); >>>>>>>>>> >>>>>>>>>> +/** >>>>>>>>>> + * vb2_core_delete_buf() - >>>>>>>>>> + * @q: pointer to &struct vb2_queue with videobuf2 queue. >>>>>>>>>> + * @index: id number of the buffer. >>>>>>>>>> + * >>>>>>>>>> + * Return: returns zero on success; an error code otherwise. >>>>>>>>>> + */ >>>>>>>>>> +int vb2_core_delete_buf(struct vb2_queue *q, unsigned int >>>>>>>>>> index); >>>>>>>>>> + >>>>>>>>>> /** >>>>>>>>>> * vb2_core_qbuf() - Queue a buffer from userspace >>>>>>>>>> * >>>>>>>>>> diff --git a/include/media/videobuf2-v4l2.h >>>>>>>>>> b/include/media/videobuf2-v4l2.h >>>>>>>>>> index 88a7a565170e..3beeb4c735f0 100644 >>>>>>>>>> --- a/include/media/videobuf2-v4l2.h >>>>>>>>>> +++ b/include/media/videobuf2-v4l2.h >>>>>>>>>> @@ -114,6 +114,17 @@ int vb2_create_bufs(struct vb2_queue *q, >>>>>>>>>> struct >>>>>>>>>> v4l2_create_buffers *create); >>>>>>>>>> */ >>>>>>>>>> int vb2_prepare_buf(struct vb2_queue *q, struct media_device >>>>>>>>>> *mdev, >>>>>>>>>> struct v4l2_buffer *b); >>>>>>>>>> +/** >>>>>>>>>> + * vb2_delete_buf() - Delete the buffer from the queue >>>>>>>>>> + * >>>>>>>>>> + * @q: pointer to &struct vb2_queue with videobuf2 >>>>>>>>>> queue. >>>>>>>>>> + * @b: buffer structure passed from userspace to >>>>>>>>>> + * &v4l2_ioctl_ops->vidioc_delete_buf handler in driver >>>>>>>>>> + * >>>>>>>>>> + * The return values from this function are intended to be >>>>>>>>>> directly >>>>>>>>>> returned >>>>>>>>>> + * from &v4l2_ioctl_ops->vidioc_delete_buf handler in driver. >>>>>>>>>> + */ >>>>>>>>>> +int vb2_delete_buf(struct vb2_queue *q, struct v4l2_buffer *b); >>>>>>>>>> >>>>>>>>>> /** >>>>>>>>>> * vb2_qbuf() - Queue a buffer from userspace >>>>>>>>>> diff --git a/include/uapi/linux/videodev2.h >>>>>>>>>> b/include/uapi/linux/videodev2.h >>>>>>>>>> index aee75eb9e686..31bba1915642 100644 >>>>>>>>>> --- a/include/uapi/linux/videodev2.h >>>>>>>>>> +++ b/include/uapi/linux/videodev2.h >>>>>>>>>> @@ -2702,6 +2702,8 @@ struct v4l2_create_buffers { >>>>>>>>>> #define VIDIOC_DBG_G_CHIP_INFO _IOWR('V', 102, struct >>>>>>>>>> v4l2_dbg_chip_info) >>>>>>>>>> >>>>>>>>>> #define VIDIOC_QUERY_EXT_CTRL _IOWR('V', 103, struct >>>>>>>>>> v4l2_query_ext_ctrl) >>>>>>>>>> +#define VIDIOC_DELETE_BUF _IOWR('V', 104, struct >>>>>>>>>> v4l2_buffer) >>>>>>>>>> + >>>>>>>>>> >>>>>>>>>> /* Reminder: when adding new ioctls please add support for >>>>>>>>>> them to >>>>>>>>>> drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */ >>>>>>>>>> -- >>>>>>>>>> 2.39.2 >>>>>>>>>> >>>>>>> -- >>>>>>> Hsia-Jun(Randy) Li >>>>>>> -- Hsia-Jun(Randy) Li ^ permalink raw reply [flat|nested] 27+ messages in thread
[parent not found: <8ca2f66e-8ff9-e885-274f-51417b581b78@synaptics.com>]
* Re: [PATCH v3 10/11] media: v4l2: Add DELETE_BUF ioctl [not found] ` <8ca2f66e-8ff9-e885-274f-51417b581b78@synaptics.com> @ 2023-07-03 11:17 ` Benjamin Gaignard 2023-07-03 15:42 ` Randy Li 0 siblings, 1 reply; 27+ messages in thread From: Benjamin Gaignard @ 2023-07-03 11:17 UTC (permalink / raw) To: Hsia-Jun Li Cc: linux-media, linux-kernel, p.zabel, ming.qian, hverkuil-cisco, gregkh, ezequiel, linux-arm-kernel, linux-mediatek, linux-arm-msm, linux-rockchip, nicolas.dufresne, linux-staging, kernel, mchehab, tfiga, m.szyprowski, ayaka Le 03/07/2023 à 13:02, Hsia-Jun Li a écrit : > > > On 7/3/23 18:35, Benjamin Gaignard wrote: >> CAUTION: Email originated externally, do not click links or open >> attachments unless you recognize the sender and know the content is >> safe. >> >> >> Le 03/07/2023 à 11:20, Hsia-Jun Li a écrit : >>> >>> On 7/3/23 16:52, Benjamin Gaignard wrote: >>>> CAUTION: Email originated externally, do not click links or open >>>> attachments unless you recognize the sender and know the content is >>>> safe. >>>> >>>> >>>> Le 03/07/2023 à 10:19, Hsia-Jun Li a écrit : >>>>> >>>>> On 7/3/23 16:12, Benjamin Gaignard wrote: >>>>>> CAUTION: Email originated externally, do not click links or open >>>>>> attachments unless you recognize the sender and know the content is >>>>>> safe. >>>>>> >>>>>> >>>>>> Le 30/06/2023 à 11:43, Hsia-Jun Li a écrit : >>>>>>> >>>>>>> On 6/27/23 16:47, Hsia-Jun Li wrote: >>>>>>>> CAUTION: Email originated externally, do not click links or open >>>>>>>> attachments unless you recognize the sender and know the >>>>>>>> content is >>>>>>>> safe. >>>>>>>> >>>>>>>> >>>>>>>> On 6/27/23 16:43, Benjamin Gaignard wrote: >>>>>>>>> CAUTION: Email originated externally, do not click links or open >>>>>>>>> attachments unless you recognize the sender and know the >>>>>>>>> content is >>>>>>>>> safe. >>>>>>>>> >>>>>>>>> >>>>>>>>> Le 27/06/2023 à 09:30, Hsia-Jun Li a écrit : >>>>>>>>>> >>>>>>>>>> On 6/22/23 21:13, Benjamin Gaignard wrote: >>>>>>>>>>> CAUTION: Email originated externally, do not click links or >>>>>>>>>>> open >>>>>>>>>>> attachments unless you recognize the sender and know the >>>>>>>>>>> content is >>>>>>>>>>> safe. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> VIDIOC_DELETE_BUF ioctl allows to delete a buffer from a queue. >>>>>>>>>>> >>>>>>>>>>> Signed-off-by: Benjamin Gaignard >>>>>>>>>>> <benjamin.gaignard@collabora.com> >>>>>>>>>>> --- >>>>>>>>>>> .../userspace-api/media/v4l/user-func.rst | 1 + >>>>>>>>>>> .../media/v4l/vidioc-delete-buf.rst | 51 >>>>>>>>>>> +++++++++++++++++++ >>>>>>>>>>> .../media/common/videobuf2/videobuf2-core.c | 33 ++++++++++++ >>>>>>>>>>> .../media/common/videobuf2/videobuf2-v4l2.c | 6 +++ >>>>>>>>>>> drivers/media/v4l2-core/v4l2-dev.c | 1 + >>>>>>>>>>> drivers/media/v4l2-core/v4l2-ioctl.c | 10 ++++ >>>>>>>>>>> include/media/v4l2-ioctl.h | 4 ++ >>>>>>>>>>> include/media/videobuf2-core.h | 9 ++++ >>>>>>>>>>> include/media/videobuf2-v4l2.h | 11 ++++ >>>>>>>>>>> include/uapi/linux/videodev2.h | 2 + >>>>>>>>>>> 10 files changed, 128 insertions(+) >>>>>>>>>>> create mode 100644 >>>>>>>>>>> Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>>>>>>> >>>>>>>>>>> diff --git >>>>>>>>>>> a/Documentation/userspace-api/media/v4l/user-func.rst >>>>>>>>>>> b/Documentation/userspace-api/media/v4l/user-func.rst >>>>>>>>>>> index 15ff0bf7bbe6..8c74016e12fd 100644 >>>>>>>>>>> --- a/Documentation/userspace-api/media/v4l/user-func.rst >>>>>>>>>>> +++ b/Documentation/userspace-api/media/v4l/user-func.rst >>>>>>>>>>> @@ -17,6 +17,7 @@ Function Reference >>>>>>>>>>> vidioc-dbg-g-chip-info >>>>>>>>>>> vidioc-dbg-g-register >>>>>>>>>>> vidioc-decoder-cmd >>>>>>>>>>> + vidioc-delete-buf >>>>>>>>>>> vidioc-dqevent >>>>>>>>>>> vidioc-dv-timings-cap >>>>>>>>>>> vidioc-encoder-cmd >>>>>>>>>>> diff --git >>>>>>>>>>> a/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>>>>>>> b/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>>>>>>> new file mode 100644 >>>>>>>>>>> index 000000000000..0e7ce58f91bc >>>>>>>>>>> --- /dev/null >>>>>>>>>>> +++ >>>>>>>>>>> b/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>>>>>>> @@ -0,0 +1,51 @@ >>>>>>>>>>> +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later >>>>>>>>>>> +.. c:namespace:: V4L >>>>>>>>>>> + >>>>>>>>>>> +.. _VIDIOC_DELETE_BUF: >>>>>>>>>>> + >>>>>>>>>>> +************************ >>>>>>>>>>> +ioctl VIDIOC_DELETE_BUF >>>>>>>>>>> +************************ >>>>>>>>>>> + >>>>>>>>>>> +Name >>>>>>>>>>> +==== >>>>>>>>>>> + >>>>>>>>>>> +VIDIOC_DELETE_BUF - Delete a buffer from a queue >>>>>>>>>>> + >>>>>>>>>>> +Synopsis >>>>>>>>>>> +======== >>>>>>>>>>> + >>>>>>>>>>> +.. c:macro:: VIDIOC_DELETE_BUF >>>>>>>>>>> + >>>>>>>>>>> +``int ioctl(int fd, VIDIOC_DELETE_BUF, struct v4l2_buffer >>>>>>>>>>> *argp)`` >>>>>>>>>>> + >>>>>>>>>>> +Arguments >>>>>>>>>>> +========= >>>>>>>>>>> + >>>>>>>>>>> +``fd`` >>>>>>>>>>> + File descriptor returned by :c:func:`open()`. >>>>>>>>>>> + >>>>>>>>>>> +``argp`` >>>>>>>>>>> + Pointer to struct :c:type:`v4l2_buffer`. >>>>>>>>>>> + >>>>>>>>>>> +Description >>>>>>>>>>> +=========== >>>>>>>>>>> + >>>>>>>>>>> +Applications can optionally call the :ref:`VIDIOC_DELETE_BUF` >>>>>>>>>>> ioctl to >>>>>>>>>>> +delete a buffer from a queue. >>>>>>>>>>> + >>>>>>>>>>> +The struct :c:type:`v4l2_buffer` structure is specified in >>>>>>>>>>> +:ref:`buffer`. >>>>>>>>>>> + >>>>>>>>>>> +Return Value >>>>>>>>>>> +============ >>>>>>>>>>> + >>>>>>>>>>> +On success 0 is returned, on error -1 and the ``errno`` >>>>>>>>>>> variable is >>>>>>>>>>> set >>>>>>>>>>> +appropriately. The generic error codes are described at the >>>>>>>>>>> +:ref:`Generic Error Codes <gen-errors>` chapter. >>>>>>>>>>> + >>>>>>>>>>> +EBUSY >>>>>>>>>>> + File I/O is in progress. >>>>>>>>>>> + >>>>>>>>>>> +EINVAL >>>>>>>>>>> + The buffer ``index`` doesn't exist in the queue. >>>>>>>>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>>>>> b/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>>>>> index 899783f67580..aa546c972c3d 100644 >>>>>>>>>>> --- a/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>>>>> +++ b/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>>>>> @@ -1637,6 +1637,39 @@ int vb2_core_prepare_buf(struct >>>>>>>>>>> vb2_queue >>>>>>>>>>> *q, >>>>>>>>>>> unsigned int index, void *pb) >>>>>>>>>>> } >>>>>>>>>>> EXPORT_SYMBOL_GPL(vb2_core_prepare_buf); >>>>>>>>>>> >>>>>>>>>>> +int vb2_core_delete_buf(struct vb2_queue *q, unsigned int >>>>>>>>>>> index) >>>>>>>>>>> +{ >>>>>>>>>>> + struct vb2_buffer *vb; >>>>>>>>>>> + >>>>>>>>>>> + vb = vb2_get_buffer(q, index); >>>>>>>>>>> + if (!vb) { >>>>>>>>>>> + dprintk(q, 1, "invalid buffer index %d\n", >>>>>>>>>>> index); >>>>>>>>>>> + return -EINVAL; >>>>>>>>>>> + } >>>>>>>>>>> + >>>>>>>>>>> + if (vb->state != VB2_BUF_STATE_DEQUEUED) { >>>>>>>>>>> + dprintk(q, 1, "can't delete non dequeued buffer >>>>>>>>>>> index >>>>>>>>>>> %d\n", index); >>>>>>>>>>> + return -EINVAL; >>>>>>>>>>> + } >>>>>>>>>>> + >>>>>>>>>> I know the driver could implement its own >>>>>>>>>> v4l2_ioctl_ops->vidioc_delete_buf() that check whether a >>>>>>>>>> buffer is >>>>>>>>>> used by the hardware as a future reference frame. >>>>>>>>>> But I think we need a flag to let the user know which buffer is >>>>>>>>>> still >>>>>>>>>> used by the hardware. >>>>>>>>>> Alternative ref case is safe, we only know it's existing when >>>>>>>>>> it is >>>>>>>>>> dequeued in current V4L2 buffer mechanism. >>>>>>>>>> While the Golden reference frame, such long term reference frame >>>>>>>>>> could >>>>>>>>>> last much longer. >>>>>>>>> >>>>>>>>> It is up to userland stack to know frames life time, it got the >>>>>>>>> information for that. >>>>>>>> >>>>>>>> That is true for the stateless codec driver. >>>>>>>> >>>>>>>> While application for stateful decoder could never do that. It >>>>>>>> also >>>>>>>> breaks what the document said: >>>>>>>> >>>>>>>> "The backing memory of |CAPTURE| buffers that are used as >>>>>>>> reference >>>>>>>> frames by the stream may be read by the hardware even after >>>>>>>> they are >>>>>>>> dequeued. Consequently, the client should avoid writing into this >>>>>>>> memory >>>>>>>> while the |CAPTURE| queue is streaming. Failure to observe this >>>>>>>> may >>>>>>>> result in corruption of decoded frames." >>>>>>>> >>>>>>>>> >>>>>>>>>>> + if (vb->planes[0].mem_priv) >>>>>>>>>>> + call_void_vb_qop(vb, buf_cleanup, vb); >>>>>>>>>>> + >>>>>>>>>>> + /* Free MMAP buffers or release USERPTR buffers */ >>>>>>>>>>> + if (q->memory == VB2_MEMORY_MMAP) >>>>>>>>>>> + __vb2_buf_mem_free(vb); >>>>>>> >>>>>>> Here is another problem for the existing application, the mmap() >>>>>>> from >>>>>>> the mmap offset or exportbuffer fd would not create a reference to >>>>>>> buffer in this step(while the exportbuffer would create one >>>>>>> itself). >>>>>>> >>>>>>> When you delete a buffer, you may not release it from its virtual >>>>>>> memory space, leaving a corrupted virtual memory space. Also this >>>>>>> behavior is right, because mmap(2) says: >>>>>>> >>>>>>> "After the mmap() call has returned, the file descriptor, fd, >>>>>>> can >>>>>>> be closed immediately without invalidating the map‐ping." >>>>>> >>>>>> Existing applications do not call DELETE_BUF ioctl and when call it >>>>>> they will be aware that the buffer is removed. >>>>>> I have done it in GStreamer: >>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__gitlab.freedesktop.org_benjamin.gaignard1_gstreamer_-2D_commit_fca0fbc934f4440693ce0ff6c8dc8a2e5f5f17d9&d=DwIDaQ&c=7dfBJ8cXbWjhc0BhImu8wVIoUFmBzj1s88r8EGyM0UY&r=P4xb2_7biqBxD4LGGPrSV6j-jf3C3xlR7PXU-mLTeZE&m=TGH9toTzGRfO5aBsfaMvGbcOw-28q6cPmpX6vScbHjpCtaLtb-RuvBvsJ0z9RvAB&s=Ufl1ccfRZf2EhnfCBvnQzRJV9CDhGxl5spe9WNECspU&e= >>>>>> >>>>>> >>>>>> >>>>> >>>>> I have read that. >>>>> >>>>> There is not a VP8 parser in Gstreamer, while a parser would not work >>>>> when deal with the secure video(userspace can't access the data >>>>> context at all). >>>>> >>>>> Besides, this adds extra work for the application for a stateful >>>>> codec >>>>> driver. The application need to parser the bitstream and track the >>>>> dpb. >>>>> >>>>> I don't mind if you could fix the nonfiction mechanism for those >>>>> non-display frame and internal reference state. >>>>> >>>>> That could be requirement for codec firmware that its driver could >>>>> support this DELETE_BUF ioctl() feature. >>>> >>>> Sorry I don't see the link with my patches here... >>>> I have work on non-secure VP9 on stateless codec. >>>> DELETE_BUF ioctl is optional and the main goal is to offer a way to >>>> applications >>>> to save memory if they know when they could delete buffers without >>>> risk. >>> >>> I try to explain why I think this design in not "complete". One >>> problem resolved, more problems would occur. >>> >>> >>> For non-secure video, those applications have worked: >>> >>> - It would break what stateful means here, application need to >>> acquire(parse) the information that driver should offer. >>> >>> Or it would break the decoding model. >>> >>> - Your Gstreamer sample code or this design won't work for AV1. >> why ? > > A frame in AV1 could be put to display order with a future frame. That > is quite different to those ITU codecs. > > Supposing a frame (we call it frame 0) without show_display frame, you > could delete it (just want to do that) once it is decoded. > > Actually, unless 7 future frames have came and parser has acknowledged > them, it is not safe to delete that frame. > > Or a show_exist that make frame 0 present. > > In my personal opinion, this feature could break many uAPI's, I have > said my worrying in the other emails. > Of course if the application delete a useful buffer it is a problem but an application issue. I have test this code using GStreamer with VP9, VP8 and HEVC on Verisilicon driver without issues. I will be happy if someone could test it on another driver and/or other userlands (like ffmpeg). >>> >>> For all the future possible secure video: >>> >>> - This feature could never be used from the current design. >>> >>> >>>> >>>>> >>>>>> >>>>>> Regards, >>>>>> Benjamin >>>>>> >>>>>>> >>>>>>>>>>> + else if (q->memory == VB2_MEMORY_DMABUF) >>>>>>>>>>> + __vb2_buf_dmabuf_put(vb); >>>>>>>>>>> + else >>>>>>>>>>> + __vb2_buf_userptr_put(vb); >>>>>>>>>>> + >>>>>>>>>>> + vb2_queue_remove_buffer(q, vb); >>>>>>>>>>> + kfree(vb); >>>>>>>>>>> + >>>>>>>>>>> + dprintk(q, 2, "buffer %d deleted\n", index); >>>>>>>>>>> + return 0; >>>>>>>>>>> +} >>>>>>>>>>> + >>>>>>>>>>> /* >>>>>>>>>>> * vb2_start_streaming() - Attempt to start streaming. >>>>>>>>>>> * @q: videobuf2 queue >>>>>>>>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>>>>>>> b/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>>>>>>> index 724135d41f7f..cea666c17b41 100644 >>>>>>>>>>> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>>>>>>> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>>>>>>> @@ -751,6 +751,12 @@ int vb2_prepare_buf(struct vb2_queue *q, >>>>>>>>>>> struct >>>>>>>>>>> media_device *mdev, >>>>>>>>>>> } >>>>>>>>>>> EXPORT_SYMBOL_GPL(vb2_prepare_buf); >>>>>>>>>>> >>>>>>>>>>> +int vb2_delete_buf(struct vb2_queue *q, struct v4l2_buffer *b) >>>>>>>>>>> +{ >>>>>>>>>>> + return vb2_core_delete_buf(q, b->index); >>>>>>>>>>> +} >>>>>>>>>>> +EXPORT_SYMBOL_GPL(vb2_delete_buf); >>>>>>>>>>> + >>>>>>>>>>> int vb2_create_bufs(struct vb2_queue *q, struct >>>>>>>>>>> v4l2_create_buffers >>>>>>>>>>> *create) >>>>>>>>>>> { >>>>>>>>>>> unsigned requested_planes = 1; >>>>>>>>>>> diff --git a/drivers/media/v4l2-core/v4l2-dev.c >>>>>>>>>>> b/drivers/media/v4l2-core/v4l2-dev.c >>>>>>>>>>> index f81279492682..80ace2e1e932 100644 >>>>>>>>>>> --- a/drivers/media/v4l2-core/v4l2-dev.c >>>>>>>>>>> +++ b/drivers/media/v4l2-core/v4l2-dev.c >>>>>>>>>>> @@ -720,6 +720,7 @@ static void determine_valid_ioctls(struct >>>>>>>>>>> video_device *vdev) >>>>>>>>>>> SET_VALID_IOCTL(ops, VIDIOC_PREPARE_BUF, >>>>>>>>>>> vidioc_prepare_buf); >>>>>>>>>>> SET_VALID_IOCTL(ops, VIDIOC_STREAMON, >>>>>>>>>>> vidioc_streamon); >>>>>>>>>>> SET_VALID_IOCTL(ops, VIDIOC_STREAMOFF, >>>>>>>>>>> vidioc_streamoff); >>>>>>>>>>> + SET_VALID_IOCTL(ops, VIDIOC_DELETE_BUF, >>>>>>>>>>> vidioc_delete_buf); >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> if (is_vid || is_vbi || is_meta) { >>>>>>>>>>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>>>>>>> b/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>>>>>>> index a858acea6547..1c737279d3ef 100644 >>>>>>>>>>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>>>>>>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>>>>>>> @@ -2156,6 +2156,15 @@ static int v4l_prepare_buf(const struct >>>>>>>>>>> v4l2_ioctl_ops *ops, >>>>>>>>>>> return ret ? ret : ops->vidioc_prepare_buf(file, >>>>>>>>>>> fh, b); >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> +static int v4l_delete_buf(const struct v4l2_ioctl_ops *ops, >>>>>>>>>>> + struct file *file, void *fh, void >>>>>>>>>>> *arg) >>>>>>>>>>> +{ >>>>>>>>>>> + struct v4l2_buffer *b = arg; >>>>>>>>>>> + int ret = check_fmt(file, b->type); >>>>>>>>>>> + >>>>>>>>>>> + return ret ? ret : ops->vidioc_delete_buf(file, fh, b); >>>>>>>>>>> +} >>>>>>>>>>> + >>>>>>>>>>> static int v4l_g_parm(const struct v4l2_ioctl_ops *ops, >>>>>>>>>>> struct file *file, void *fh, >>>>>>>>>>> void >>>>>>>>>>> *arg) >>>>>>>>>>> { >>>>>>>>>>> @@ -2905,6 +2914,7 @@ static const struct v4l2_ioctl_info >>>>>>>>>>> v4l2_ioctls[] = { >>>>>>>>>>> IOCTL_INFO(VIDIOC_ENUM_FREQ_BANDS, >>>>>>>>>>> v4l_enum_freq_bands, >>>>>>>>>>> v4l_print_freq_band, 0), >>>>>>>>>>> IOCTL_INFO(VIDIOC_DBG_G_CHIP_INFO, >>>>>>>>>>> v4l_dbg_g_chip_info, >>>>>>>>>>> v4l_print_dbg_chip_info, INFO_FL_CLEAR(v4l2_dbg_chip_info, >>>>>>>>>>> match)), >>>>>>>>>>> IOCTL_INFO(VIDIOC_QUERY_EXT_CTRL, v4l_query_ext_ctrl, >>>>>>>>>>> v4l_print_query_ext_ctrl, INFO_FL_CTRL | >>>>>>>>>>> INFO_FL_CLEAR(v4l2_query_ext_ctrl, id)), >>>>>>>>>>> + IOCTL_INFO(VIDIOC_DELETE_BUF, v4l_delete_buf, >>>>>>>>>>> v4l_print_buffer, INFO_FL_QUEUE), >>>>>>>>>>> }; >>>>>>>>>>> #define V4L2_IOCTLS ARRAY_SIZE(v4l2_ioctls) >>>>>>>>>>> >>>>>>>>>>> diff --git a/include/media/v4l2-ioctl.h >>>>>>>>>>> b/include/media/v4l2-ioctl.h >>>>>>>>>>> index edb733f21604..2f232ed884c7 100644 >>>>>>>>>>> --- a/include/media/v4l2-ioctl.h >>>>>>>>>>> +++ b/include/media/v4l2-ioctl.h >>>>>>>>>>> @@ -163,6 +163,8 @@ struct v4l2_fh; >>>>>>>>>>> * :ref:`VIDIOC_CREATE_BUFS <vidioc_create_bufs>` ioctl >>>>>>>>>>> * @vidioc_prepare_buf: pointer to the function that >>>>>>>>>>> implements >>>>>>>>>>> * :ref:`VIDIOC_PREPARE_BUF <vidioc_prepare_buf>` ioctl >>>>>>>>>>> + * @vidioc_delete_buf: pointer to the function that implements >>>>>>>>>>> + * :ref:`VIDIOC_DELETE_BUF <vidioc_delete_buf>` ioctl >>>>>>>>>>> * @vidioc_overlay: pointer to the function that implements >>>>>>>>>>> * :ref:`VIDIOC_OVERLAY <vidioc_overlay>` ioctl >>>>>>>>>>> * @vidioc_g_fbuf: pointer to the function that implements >>>>>>>>>>> @@ -422,6 +424,8 @@ struct v4l2_ioctl_ops { >>>>>>>>>>> struct v4l2_create_buffers >>>>>>>>>>> *b); >>>>>>>>>>> int (*vidioc_prepare_buf)(struct file *file, void *fh, >>>>>>>>>>> struct v4l2_buffer *b); >>>>>>>>>>> + int (*vidioc_delete_buf)(struct file *file, void *fh, >>>>>>>>>>> + struct v4l2_buffer *b); >>>>>>>>>>> >>>>>>>>>>> int (*vidioc_overlay)(struct file *file, void *fh, >>>>>>>>>>> unsigned >>>>>>>>>>> int i); >>>>>>>>>>> int (*vidioc_g_fbuf)(struct file *file, void *fh, >>>>>>>>>>> diff --git a/include/media/videobuf2-core.h >>>>>>>>>>> b/include/media/videobuf2-core.h >>>>>>>>>>> index 080b783d608d..0f9e68f76b77 100644 >>>>>>>>>>> --- a/include/media/videobuf2-core.h >>>>>>>>>>> +++ b/include/media/videobuf2-core.h >>>>>>>>>>> @@ -840,6 +840,15 @@ int vb2_core_create_bufs(struct vb2_queue >>>>>>>>>>> *q, >>>>>>>>>>> enum vb2_memory memory, >>>>>>>>>>> */ >>>>>>>>>>> int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int >>>>>>>>>>> index, >>>>>>>>>>> void *pb); >>>>>>>>>>> >>>>>>>>>>> +/** >>>>>>>>>>> + * vb2_core_delete_buf() - >>>>>>>>>>> + * @q: pointer to &struct vb2_queue with videobuf2 queue. >>>>>>>>>>> + * @index: id number of the buffer. >>>>>>>>>>> + * >>>>>>>>>>> + * Return: returns zero on success; an error code otherwise. >>>>>>>>>>> + */ >>>>>>>>>>> +int vb2_core_delete_buf(struct vb2_queue *q, unsigned int >>>>>>>>>>> index); >>>>>>>>>>> + >>>>>>>>>>> /** >>>>>>>>>>> * vb2_core_qbuf() - Queue a buffer from userspace >>>>>>>>>>> * >>>>>>>>>>> diff --git a/include/media/videobuf2-v4l2.h >>>>>>>>>>> b/include/media/videobuf2-v4l2.h >>>>>>>>>>> index 88a7a565170e..3beeb4c735f0 100644 >>>>>>>>>>> --- a/include/media/videobuf2-v4l2.h >>>>>>>>>>> +++ b/include/media/videobuf2-v4l2.h >>>>>>>>>>> @@ -114,6 +114,17 @@ int vb2_create_bufs(struct vb2_queue *q, >>>>>>>>>>> struct >>>>>>>>>>> v4l2_create_buffers *create); >>>>>>>>>>> */ >>>>>>>>>>> int vb2_prepare_buf(struct vb2_queue *q, struct media_device >>>>>>>>>>> *mdev, >>>>>>>>>>> struct v4l2_buffer *b); >>>>>>>>>>> +/** >>>>>>>>>>> + * vb2_delete_buf() - Delete the buffer from the queue >>>>>>>>>>> + * >>>>>>>>>>> + * @q: pointer to &struct vb2_queue with videobuf2 >>>>>>>>>>> queue. >>>>>>>>>>> + * @b: buffer structure passed from userspace to >>>>>>>>>>> + * &v4l2_ioctl_ops->vidioc_delete_buf handler in driver >>>>>>>>>>> + * >>>>>>>>>>> + * The return values from this function are intended to be >>>>>>>>>>> directly >>>>>>>>>>> returned >>>>>>>>>>> + * from &v4l2_ioctl_ops->vidioc_delete_buf handler in driver. >>>>>>>>>>> + */ >>>>>>>>>>> +int vb2_delete_buf(struct vb2_queue *q, struct v4l2_buffer >>>>>>>>>>> *b); >>>>>>>>>>> >>>>>>>>>>> /** >>>>>>>>>>> * vb2_qbuf() - Queue a buffer from userspace >>>>>>>>>>> diff --git a/include/uapi/linux/videodev2.h >>>>>>>>>>> b/include/uapi/linux/videodev2.h >>>>>>>>>>> index aee75eb9e686..31bba1915642 100644 >>>>>>>>>>> --- a/include/uapi/linux/videodev2.h >>>>>>>>>>> +++ b/include/uapi/linux/videodev2.h >>>>>>>>>>> @@ -2702,6 +2702,8 @@ struct v4l2_create_buffers { >>>>>>>>>>> #define VIDIOC_DBG_G_CHIP_INFO _IOWR('V', 102, struct >>>>>>>>>>> v4l2_dbg_chip_info) >>>>>>>>>>> >>>>>>>>>>> #define VIDIOC_QUERY_EXT_CTRL _IOWR('V', 103, struct >>>>>>>>>>> v4l2_query_ext_ctrl) >>>>>>>>>>> +#define VIDIOC_DELETE_BUF _IOWR('V', 104, struct >>>>>>>>>>> v4l2_buffer) >>>>>>>>>>> + >>>>>>>>>>> >>>>>>>>>>> /* Reminder: when adding new ioctls please add support for >>>>>>>>>>> them to >>>>>>>>>>> drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */ >>>>>>>>>>> -- >>>>>>>>>>> 2.39.2 >>>>>>>>>>> >>>>>>>> -- >>>>>>>> Hsia-Jun(Randy) Li >>>>>>>> > -- > Hsia-Jun(Randy) Li ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v3 10/11] media: v4l2: Add DELETE_BUF ioctl 2023-07-03 11:17 ` Benjamin Gaignard @ 2023-07-03 15:42 ` Randy Li 0 siblings, 0 replies; 27+ messages in thread From: Randy Li @ 2023-07-03 15:42 UTC (permalink / raw) To: Benjamin Gaignard Cc: linux-media, Hsia-Jun Li, linux-kernel, p.zabel, ming.qian, hverkuil-cisco, gregkh, ezequiel, linux-arm-kernel, linux-mediatek, linux-arm-msm, linux-rockchip, nicolas.dufresne, linux-staging, kernel, mchehab, tfiga, m.szyprowski On 2023/7/3 19:17, Benjamin Gaignard wrote: > > Le 03/07/2023 à 13:02, Hsia-Jun Li a écrit : >> >> >> On 7/3/23 18:35, Benjamin Gaignard wrote: >>> CAUTION: Email originated externally, do not click links or open >>> attachments unless you recognize the sender and know the content is >>> safe. >>> >>> >>> Le 03/07/2023 à 11:20, Hsia-Jun Li a écrit : >>>> >>>> On 7/3/23 16:52, Benjamin Gaignard wrote: >>>>> CAUTION: Email originated externally, do not click links or open >>>>> attachments unless you recognize the sender and know the content is >>>>> safe. >>>>> >>>>> >>>>> Le 03/07/2023 à 10:19, Hsia-Jun Li a écrit : >>>>>> >>>>>> On 7/3/23 16:12, Benjamin Gaignard wrote: >>>>>>> CAUTION: Email originated externally, do not click links or open >>>>>>> attachments unless you recognize the sender and know the content is >>>>>>> safe. >>>>>>> >>>>>>> >>>>>>> Le 30/06/2023 à 11:43, Hsia-Jun Li a écrit : >>>>>>>> >>>>>>>> On 6/27/23 16:47, Hsia-Jun Li wrote: >>>>>>>>> CAUTION: Email originated externally, do not click links or open >>>>>>>>> attachments unless you recognize the sender and know the >>>>>>>>> content is >>>>>>>>> safe. >>>>>>>>> >>>>>>>>> >>>>>>>>> On 6/27/23 16:43, Benjamin Gaignard wrote: >>>>>>>>>> CAUTION: Email originated externally, do not click links or open >>>>>>>>>> attachments unless you recognize the sender and know the >>>>>>>>>> content is >>>>>>>>>> safe. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Le 27/06/2023 à 09:30, Hsia-Jun Li a écrit : >>>>>>>>>>> >>>>>>>>>>> On 6/22/23 21:13, Benjamin Gaignard wrote: >>>>>>>>>>>> CAUTION: Email originated externally, do not click links or >>>>>>>>>>>> open >>>>>>>>>>>> attachments unless you recognize the sender and know the >>>>>>>>>>>> content is >>>>>>>>>>>> safe. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> VIDIOC_DELETE_BUF ioctl allows to delete a buffer from a >>>>>>>>>>>> queue. >>>>>>>>>>>> >>>>>>>>>>>> Signed-off-by: Benjamin Gaignard >>>>>>>>>>>> <benjamin.gaignard@collabora.com> >>>>>>>>>>>> --- >>>>>>>>>>>> .../userspace-api/media/v4l/user-func.rst | 1 + >>>>>>>>>>>> .../media/v4l/vidioc-delete-buf.rst | 51 >>>>>>>>>>>> +++++++++++++++++++ >>>>>>>>>>>> .../media/common/videobuf2/videobuf2-core.c | 33 ++++++++++++ >>>>>>>>>>>> .../media/common/videobuf2/videobuf2-v4l2.c | 6 +++ >>>>>>>>>>>> drivers/media/v4l2-core/v4l2-dev.c | 1 + >>>>>>>>>>>> drivers/media/v4l2-core/v4l2-ioctl.c | 10 ++++ >>>>>>>>>>>> include/media/v4l2-ioctl.h | 4 ++ >>>>>>>>>>>> include/media/videobuf2-core.h | 9 ++++ >>>>>>>>>>>> include/media/videobuf2-v4l2.h | 11 ++++ >>>>>>>>>>>> include/uapi/linux/videodev2.h | 2 + >>>>>>>>>>>> 10 files changed, 128 insertions(+) >>>>>>>>>>>> create mode 100644 >>>>>>>>>>>> Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>>>>>>>> >>>>>>>>>>>> diff --git >>>>>>>>>>>> a/Documentation/userspace-api/media/v4l/user-func.rst >>>>>>>>>>>> b/Documentation/userspace-api/media/v4l/user-func.rst >>>>>>>>>>>> index 15ff0bf7bbe6..8c74016e12fd 100644 >>>>>>>>>>>> --- a/Documentation/userspace-api/media/v4l/user-func.rst >>>>>>>>>>>> +++ b/Documentation/userspace-api/media/v4l/user-func.rst >>>>>>>>>>>> @@ -17,6 +17,7 @@ Function Reference >>>>>>>>>>>> vidioc-dbg-g-chip-info >>>>>>>>>>>> vidioc-dbg-g-register >>>>>>>>>>>> vidioc-decoder-cmd >>>>>>>>>>>> + vidioc-delete-buf >>>>>>>>>>>> vidioc-dqevent >>>>>>>>>>>> vidioc-dv-timings-cap >>>>>>>>>>>> vidioc-encoder-cmd >>>>>>>>>>>> diff --git >>>>>>>>>>>> a/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>>>>>>>> b/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>>>>>>>> new file mode 100644 >>>>>>>>>>>> index 000000000000..0e7ce58f91bc >>>>>>>>>>>> --- /dev/null >>>>>>>>>>>> +++ >>>>>>>>>>>> b/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>>>>>>>> @@ -0,0 +1,51 @@ >>>>>>>>>>>> +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later >>>>>>>>>>>> +.. c:namespace:: V4L >>>>>>>>>>>> + >>>>>>>>>>>> +.. _VIDIOC_DELETE_BUF: >>>>>>>>>>>> + >>>>>>>>>>>> +************************ >>>>>>>>>>>> +ioctl VIDIOC_DELETE_BUF >>>>>>>>>>>> +************************ >>>>>>>>>>>> + >>>>>>>>>>>> +Name >>>>>>>>>>>> +==== >>>>>>>>>>>> + >>>>>>>>>>>> +VIDIOC_DELETE_BUF - Delete a buffer from a queue >>>>>>>>>>>> + >>>>>>>>>>>> +Synopsis >>>>>>>>>>>> +======== >>>>>>>>>>>> + >>>>>>>>>>>> +.. c:macro:: VIDIOC_DELETE_BUF >>>>>>>>>>>> + >>>>>>>>>>>> +``int ioctl(int fd, VIDIOC_DELETE_BUF, struct v4l2_buffer >>>>>>>>>>>> *argp)`` >>>>>>>>>>>> + >>>>>>>>>>>> +Arguments >>>>>>>>>>>> +========= >>>>>>>>>>>> + >>>>>>>>>>>> +``fd`` >>>>>>>>>>>> + File descriptor returned by :c:func:`open()`. >>>>>>>>>>>> + >>>>>>>>>>>> +``argp`` >>>>>>>>>>>> + Pointer to struct :c:type:`v4l2_buffer`. >>>>>>>>>>>> + >>>>>>>>>>>> +Description >>>>>>>>>>>> +=========== >>>>>>>>>>>> + >>>>>>>>>>>> +Applications can optionally call the :ref:`VIDIOC_DELETE_BUF` >>>>>>>>>>>> ioctl to >>>>>>>>>>>> +delete a buffer from a queue. >>>>>>>>>>>> + >>>>>>>>>>>> +The struct :c:type:`v4l2_buffer` structure is specified in >>>>>>>>>>>> +:ref:`buffer`. >>>>>>>>>>>> + >>>>>>>>>>>> +Return Value >>>>>>>>>>>> +============ >>>>>>>>>>>> + >>>>>>>>>>>> +On success 0 is returned, on error -1 and the ``errno`` >>>>>>>>>>>> variable is >>>>>>>>>>>> set >>>>>>>>>>>> +appropriately. The generic error codes are described at the >>>>>>>>>>>> +:ref:`Generic Error Codes <gen-errors>` chapter. >>>>>>>>>>>> + >>>>>>>>>>>> +EBUSY >>>>>>>>>>>> + File I/O is in progress. >>>>>>>>>>>> + >>>>>>>>>>>> +EINVAL >>>>>>>>>>>> + The buffer ``index`` doesn't exist in the queue. >>>>>>>>>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>>>>>> b/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>>>>>> index 899783f67580..aa546c972c3d 100644 >>>>>>>>>>>> --- a/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>>>>>> +++ b/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>>>>>> @@ -1637,6 +1637,39 @@ int vb2_core_prepare_buf(struct >>>>>>>>>>>> vb2_queue >>>>>>>>>>>> *q, >>>>>>>>>>>> unsigned int index, void *pb) >>>>>>>>>>>> } >>>>>>>>>>>> EXPORT_SYMBOL_GPL(vb2_core_prepare_buf); >>>>>>>>>>>> >>>>>>>>>>>> +int vb2_core_delete_buf(struct vb2_queue *q, unsigned int >>>>>>>>>>>> index) >>>>>>>>>>>> +{ >>>>>>>>>>>> + struct vb2_buffer *vb; >>>>>>>>>>>> + >>>>>>>>>>>> + vb = vb2_get_buffer(q, index); >>>>>>>>>>>> + if (!vb) { >>>>>>>>>>>> + dprintk(q, 1, "invalid buffer index %d\n", >>>>>>>>>>>> index); >>>>>>>>>>>> + return -EINVAL; >>>>>>>>>>>> + } >>>>>>>>>>>> + >>>>>>>>>>>> + if (vb->state != VB2_BUF_STATE_DEQUEUED) { >>>>>>>>>>>> + dprintk(q, 1, "can't delete non dequeued >>>>>>>>>>>> buffer >>>>>>>>>>>> index >>>>>>>>>>>> %d\n", index); >>>>>>>>>>>> + return -EINVAL; >>>>>>>>>>>> + } >>>>>>>>>>>> + >>>>>>>>>>> I know the driver could implement its own >>>>>>>>>>> v4l2_ioctl_ops->vidioc_delete_buf() that check whether a >>>>>>>>>>> buffer is >>>>>>>>>>> used by the hardware as a future reference frame. >>>>>>>>>>> But I think we need a flag to let the user know which buffer is >>>>>>>>>>> still >>>>>>>>>>> used by the hardware. >>>>>>>>>>> Alternative ref case is safe, we only know it's existing when >>>>>>>>>>> it is >>>>>>>>>>> dequeued in current V4L2 buffer mechanism. >>>>>>>>>>> While the Golden reference frame, such long term reference >>>>>>>>>>> frame >>>>>>>>>>> could >>>>>>>>>>> last much longer. >>>>>>>>>> >>>>>>>>>> It is up to userland stack to know frames life time, it got the >>>>>>>>>> information for that. >>>>>>>>> >>>>>>>>> That is true for the stateless codec driver. >>>>>>>>> >>>>>>>>> While application for stateful decoder could never do that. It >>>>>>>>> also >>>>>>>>> breaks what the document said: >>>>>>>>> >>>>>>>>> "The backing memory of |CAPTURE| buffers that are used as >>>>>>>>> reference >>>>>>>>> frames by the stream may be read by the hardware even after >>>>>>>>> they are >>>>>>>>> dequeued. Consequently, the client should avoid writing into this >>>>>>>>> memory >>>>>>>>> while the |CAPTURE| queue is streaming. Failure to observe >>>>>>>>> this may >>>>>>>>> result in corruption of decoded frames." >>>>>>>>> >>>>>>>>>> >>>>>>>>>>>> + if (vb->planes[0].mem_priv) >>>>>>>>>>>> + call_void_vb_qop(vb, buf_cleanup, vb); >>>>>>>>>>>> + >>>>>>>>>>>> + /* Free MMAP buffers or release USERPTR buffers */ >>>>>>>>>>>> + if (q->memory == VB2_MEMORY_MMAP) >>>>>>>>>>>> + __vb2_buf_mem_free(vb); >>>>>>>> >>>>>>>> Here is another problem for the existing application, the >>>>>>>> mmap() from >>>>>>>> the mmap offset or exportbuffer fd would not create a reference to >>>>>>>> buffer in this step(while the exportbuffer would create one >>>>>>>> itself). >>>>>>>> >>>>>>>> When you delete a buffer, you may not release it from its virtual >>>>>>>> memory space, leaving a corrupted virtual memory space. Also this >>>>>>>> behavior is right, because mmap(2) says: >>>>>>>> >>>>>>>> "After the mmap() call has returned, the file descriptor, >>>>>>>> fd, can >>>>>>>> be closed immediately without invalidating the map‐ping." >>>>>>> >>>>>>> Existing applications do not call DELETE_BUF ioctl and when call it >>>>>>> they will be aware that the buffer is removed. >>>>>>> I have done it in GStreamer: >>>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__gitlab.freedesktop.org_benjamin.gaignard1_gstreamer_-2D_commit_fca0fbc934f4440693ce0ff6c8dc8a2e5f5f17d9&d=DwIDaQ&c=7dfBJ8cXbWjhc0BhImu8wVIoUFmBzj1s88r8EGyM0UY&r=P4xb2_7biqBxD4LGGPrSV6j-jf3C3xlR7PXU-mLTeZE&m=TGH9toTzGRfO5aBsfaMvGbcOw-28q6cPmpX6vScbHjpCtaLtb-RuvBvsJ0z9RvAB&s=Ufl1ccfRZf2EhnfCBvnQzRJV9CDhGxl5spe9WNECspU&e= >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> I have read that. >>>>>> >>>>>> There is not a VP8 parser in Gstreamer, while a parser would not >>>>>> work >>>>>> when deal with the secure video(userspace can't access the data >>>>>> context at all). >>>>>> >>>>>> Besides, this adds extra work for the application for a stateful >>>>>> codec >>>>>> driver. The application need to parser the bitstream and track >>>>>> the dpb. >>>>>> >>>>>> I don't mind if you could fix the nonfiction mechanism for those >>>>>> non-display frame and internal reference state. >>>>>> >>>>>> That could be requirement for codec firmware that its driver could >>>>>> support this DELETE_BUF ioctl() feature. >>>>> >>>>> Sorry I don't see the link with my patches here... >>>>> I have work on non-secure VP9 on stateless codec. >>>>> DELETE_BUF ioctl is optional and the main goal is to offer a way to >>>>> applications >>>>> to save memory if they know when they could delete buffers without >>>>> risk. >>>> >>>> I try to explain why I think this design in not "complete". One >>>> problem resolved, more problems would occur. >>>> >>>> >>>> For non-secure video, those applications have worked: >>>> >>>> - It would break what stateful means here, application need to >>>> acquire(parse) the information that driver should offer. >>>> >>>> Or it would break the decoding model. >>>> >>>> - Your Gstreamer sample code or this design won't work for AV1. >>> why ? >> >> A frame in AV1 could be put to display order with a future frame. >> That is quite different to those ITU codecs. >> >> Supposing a frame (we call it frame 0) without show_display frame, >> you could delete it (just want to do that) once it is decoded. >> >> Actually, unless 7 future frames have came and parser has >> acknowledged them, it is not safe to delete that frame. >> >> Or a show_exist that make frame 0 present. >> >> In my personal opinion, this feature could break many uAPI's, I have >> said my worrying in the other emails. >> > Of course if the application delete a useful buffer it is a problem > but an application issue. I don't think this feature is only applied to stateless codec. Stateful codec could reclaim the previous memory as well. The implementation now just leave all those problems to the conscience of the application, no protection from the kernel which could crash the hardware easily. I am not against on this feature, I just wish it would be more safe and more complete. > I have test this code using GStreamer with VP9, VP8 and HEVC on > Verisilicon driver without issues. Even with the secure decoding, at least someone should offer the DPB info or we just don't need this. I am sure stateless decoder could work well with that despite the performance impact on buffer looking up. > > I will be happy if someone could test it on another driver and/or > other userlands (like ffmpeg). > >>>> >>>> For all the future possible secure video: >>>> >>>> - This feature could never be used from the current design. >>>> >>>> >>>>> >>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> Benjamin >>>>>>> >>>>>>>> >>>>>>>>>>>> + else if (q->memory == VB2_MEMORY_DMABUF) >>>>>>>>>>>> + __vb2_buf_dmabuf_put(vb); >>>>>>>>>>>> + else >>>>>>>>>>>> + __vb2_buf_userptr_put(vb); >>>>>>>>>>>> + >>>>>>>>>>>> + vb2_queue_remove_buffer(q, vb); >>>>>>>>>>>> + kfree(vb); >>>>>>>>>>>> + >>>>>>>>>>>> + dprintk(q, 2, "buffer %d deleted\n", index); >>>>>>>>>>>> + return 0; >>>>>>>>>>>> +} >>>>>>>>>>>> + >>>>>>>>>>>> /* >>>>>>>>>>>> * vb2_start_streaming() - Attempt to start streaming. >>>>>>>>>>>> * @q: videobuf2 queue >>>>>>>>>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>>>>>>>> b/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>>>>>>>> index 724135d41f7f..cea666c17b41 100644 >>>>>>>>>>>> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>>>>>>>> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>>>>>>>> @@ -751,6 +751,12 @@ int vb2_prepare_buf(struct vb2_queue *q, >>>>>>>>>>>> struct >>>>>>>>>>>> media_device *mdev, >>>>>>>>>>>> } >>>>>>>>>>>> EXPORT_SYMBOL_GPL(vb2_prepare_buf); >>>>>>>>>>>> >>>>>>>>>>>> +int vb2_delete_buf(struct vb2_queue *q, struct v4l2_buffer >>>>>>>>>>>> *b) >>>>>>>>>>>> +{ >>>>>>>>>>>> + return vb2_core_delete_buf(q, b->index); >>>>>>>>>>>> +} >>>>>>>>>>>> +EXPORT_SYMBOL_GPL(vb2_delete_buf); >>>>>>>>>>>> + >>>>>>>>>>>> int vb2_create_bufs(struct vb2_queue *q, struct >>>>>>>>>>>> v4l2_create_buffers >>>>>>>>>>>> *create) >>>>>>>>>>>> { >>>>>>>>>>>> unsigned requested_planes = 1; >>>>>>>>>>>> diff --git a/drivers/media/v4l2-core/v4l2-dev.c >>>>>>>>>>>> b/drivers/media/v4l2-core/v4l2-dev.c >>>>>>>>>>>> index f81279492682..80ace2e1e932 100644 >>>>>>>>>>>> --- a/drivers/media/v4l2-core/v4l2-dev.c >>>>>>>>>>>> +++ b/drivers/media/v4l2-core/v4l2-dev.c >>>>>>>>>>>> @@ -720,6 +720,7 @@ static void determine_valid_ioctls(struct >>>>>>>>>>>> video_device *vdev) >>>>>>>>>>>> SET_VALID_IOCTL(ops, VIDIOC_PREPARE_BUF, >>>>>>>>>>>> vidioc_prepare_buf); >>>>>>>>>>>> SET_VALID_IOCTL(ops, VIDIOC_STREAMON, >>>>>>>>>>>> vidioc_streamon); >>>>>>>>>>>> SET_VALID_IOCTL(ops, VIDIOC_STREAMOFF, >>>>>>>>>>>> vidioc_streamoff); >>>>>>>>>>>> + SET_VALID_IOCTL(ops, VIDIOC_DELETE_BUF, >>>>>>>>>>>> vidioc_delete_buf); >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> if (is_vid || is_vbi || is_meta) { >>>>>>>>>>>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>>>>>>>> b/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>>>>>>>> index a858acea6547..1c737279d3ef 100644 >>>>>>>>>>>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>>>>>>>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>>>>>>>> @@ -2156,6 +2156,15 @@ static int v4l_prepare_buf(const struct >>>>>>>>>>>> v4l2_ioctl_ops *ops, >>>>>>>>>>>> return ret ? ret : ops->vidioc_prepare_buf(file, >>>>>>>>>>>> fh, b); >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> +static int v4l_delete_buf(const struct v4l2_ioctl_ops *ops, >>>>>>>>>>>> + struct file *file, void *fh, void >>>>>>>>>>>> *arg) >>>>>>>>>>>> +{ >>>>>>>>>>>> + struct v4l2_buffer *b = arg; >>>>>>>>>>>> + int ret = check_fmt(file, b->type); >>>>>>>>>>>> + >>>>>>>>>>>> + return ret ? ret : ops->vidioc_delete_buf(file, fh, >>>>>>>>>>>> b); >>>>>>>>>>>> +} >>>>>>>>>>>> + >>>>>>>>>>>> static int v4l_g_parm(const struct v4l2_ioctl_ops *ops, >>>>>>>>>>>> struct file *file, void *fh, >>>>>>>>>>>> void >>>>>>>>>>>> *arg) >>>>>>>>>>>> { >>>>>>>>>>>> @@ -2905,6 +2914,7 @@ static const struct v4l2_ioctl_info >>>>>>>>>>>> v4l2_ioctls[] = { >>>>>>>>>>>> IOCTL_INFO(VIDIOC_ENUM_FREQ_BANDS, >>>>>>>>>>>> v4l_enum_freq_bands, >>>>>>>>>>>> v4l_print_freq_band, 0), >>>>>>>>>>>> IOCTL_INFO(VIDIOC_DBG_G_CHIP_INFO, >>>>>>>>>>>> v4l_dbg_g_chip_info, >>>>>>>>>>>> v4l_print_dbg_chip_info, INFO_FL_CLEAR(v4l2_dbg_chip_info, >>>>>>>>>>>> match)), >>>>>>>>>>>> IOCTL_INFO(VIDIOC_QUERY_EXT_CTRL, v4l_query_ext_ctrl, >>>>>>>>>>>> v4l_print_query_ext_ctrl, INFO_FL_CTRL | >>>>>>>>>>>> INFO_FL_CLEAR(v4l2_query_ext_ctrl, id)), >>>>>>>>>>>> + IOCTL_INFO(VIDIOC_DELETE_BUF, v4l_delete_buf, >>>>>>>>>>>> v4l_print_buffer, INFO_FL_QUEUE), >>>>>>>>>>>> }; >>>>>>>>>>>> #define V4L2_IOCTLS ARRAY_SIZE(v4l2_ioctls) >>>>>>>>>>>> >>>>>>>>>>>> diff --git a/include/media/v4l2-ioctl.h >>>>>>>>>>>> b/include/media/v4l2-ioctl.h >>>>>>>>>>>> index edb733f21604..2f232ed884c7 100644 >>>>>>>>>>>> --- a/include/media/v4l2-ioctl.h >>>>>>>>>>>> +++ b/include/media/v4l2-ioctl.h >>>>>>>>>>>> @@ -163,6 +163,8 @@ struct v4l2_fh; >>>>>>>>>>>> * :ref:`VIDIOC_CREATE_BUFS <vidioc_create_bufs>` ioctl >>>>>>>>>>>> * @vidioc_prepare_buf: pointer to the function that >>>>>>>>>>>> implements >>>>>>>>>>>> * :ref:`VIDIOC_PREPARE_BUF <vidioc_prepare_buf>` ioctl >>>>>>>>>>>> + * @vidioc_delete_buf: pointer to the function that >>>>>>>>>>>> implements >>>>>>>>>>>> + * :ref:`VIDIOC_DELETE_BUF <vidioc_delete_buf>` ioctl >>>>>>>>>>>> * @vidioc_overlay: pointer to the function that implements >>>>>>>>>>>> * :ref:`VIDIOC_OVERLAY <vidioc_overlay>` ioctl >>>>>>>>>>>> * @vidioc_g_fbuf: pointer to the function that implements >>>>>>>>>>>> @@ -422,6 +424,8 @@ struct v4l2_ioctl_ops { >>>>>>>>>>>> struct v4l2_create_buffers >>>>>>>>>>>> *b); >>>>>>>>>>>> int (*vidioc_prepare_buf)(struct file *file, void >>>>>>>>>>>> *fh, >>>>>>>>>>>> struct v4l2_buffer *b); >>>>>>>>>>>> + int (*vidioc_delete_buf)(struct file *file, void *fh, >>>>>>>>>>>> + struct v4l2_buffer *b); >>>>>>>>>>>> >>>>>>>>>>>> int (*vidioc_overlay)(struct file *file, void *fh, >>>>>>>>>>>> unsigned >>>>>>>>>>>> int i); >>>>>>>>>>>> int (*vidioc_g_fbuf)(struct file *file, void *fh, >>>>>>>>>>>> diff --git a/include/media/videobuf2-core.h >>>>>>>>>>>> b/include/media/videobuf2-core.h >>>>>>>>>>>> index 080b783d608d..0f9e68f76b77 100644 >>>>>>>>>>>> --- a/include/media/videobuf2-core.h >>>>>>>>>>>> +++ b/include/media/videobuf2-core.h >>>>>>>>>>>> @@ -840,6 +840,15 @@ int vb2_core_create_bufs(struct vb2_queue >>>>>>>>>>>> *q, >>>>>>>>>>>> enum vb2_memory memory, >>>>>>>>>>>> */ >>>>>>>>>>>> int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int >>>>>>>>>>>> index, >>>>>>>>>>>> void *pb); >>>>>>>>>>>> >>>>>>>>>>>> +/** >>>>>>>>>>>> + * vb2_core_delete_buf() - >>>>>>>>>>>> + * @q: pointer to &struct vb2_queue with videobuf2 queue. >>>>>>>>>>>> + * @index: id number of the buffer. >>>>>>>>>>>> + * >>>>>>>>>>>> + * Return: returns zero on success; an error code otherwise. >>>>>>>>>>>> + */ >>>>>>>>>>>> +int vb2_core_delete_buf(struct vb2_queue *q, unsigned int >>>>>>>>>>>> index); >>>>>>>>>>>> + >>>>>>>>>>>> /** >>>>>>>>>>>> * vb2_core_qbuf() - Queue a buffer from userspace >>>>>>>>>>>> * >>>>>>>>>>>> diff --git a/include/media/videobuf2-v4l2.h >>>>>>>>>>>> b/include/media/videobuf2-v4l2.h >>>>>>>>>>>> index 88a7a565170e..3beeb4c735f0 100644 >>>>>>>>>>>> --- a/include/media/videobuf2-v4l2.h >>>>>>>>>>>> +++ b/include/media/videobuf2-v4l2.h >>>>>>>>>>>> @@ -114,6 +114,17 @@ int vb2_create_bufs(struct vb2_queue *q, >>>>>>>>>>>> struct >>>>>>>>>>>> v4l2_create_buffers *create); >>>>>>>>>>>> */ >>>>>>>>>>>> int vb2_prepare_buf(struct vb2_queue *q, struct media_device >>>>>>>>>>>> *mdev, >>>>>>>>>>>> struct v4l2_buffer *b); >>>>>>>>>>>> +/** >>>>>>>>>>>> + * vb2_delete_buf() - Delete the buffer from the queue >>>>>>>>>>>> + * >>>>>>>>>>>> + * @q: pointer to &struct vb2_queue with videobuf2 >>>>>>>>>>>> queue. >>>>>>>>>>>> + * @b: buffer structure passed from userspace to >>>>>>>>>>>> + * &v4l2_ioctl_ops->vidioc_delete_buf handler in driver >>>>>>>>>>>> + * >>>>>>>>>>>> + * The return values from this function are intended to be >>>>>>>>>>>> directly >>>>>>>>>>>> returned >>>>>>>>>>>> + * from &v4l2_ioctl_ops->vidioc_delete_buf handler in driver. >>>>>>>>>>>> + */ >>>>>>>>>>>> +int vb2_delete_buf(struct vb2_queue *q, struct v4l2_buffer >>>>>>>>>>>> *b); >>>>>>>>>>>> >>>>>>>>>>>> /** >>>>>>>>>>>> * vb2_qbuf() - Queue a buffer from userspace >>>>>>>>>>>> diff --git a/include/uapi/linux/videodev2.h >>>>>>>>>>>> b/include/uapi/linux/videodev2.h >>>>>>>>>>>> index aee75eb9e686..31bba1915642 100644 >>>>>>>>>>>> --- a/include/uapi/linux/videodev2.h >>>>>>>>>>>> +++ b/include/uapi/linux/videodev2.h >>>>>>>>>>>> @@ -2702,6 +2702,8 @@ struct v4l2_create_buffers { >>>>>>>>>>>> #define VIDIOC_DBG_G_CHIP_INFO _IOWR('V', 102, struct >>>>>>>>>>>> v4l2_dbg_chip_info) >>>>>>>>>>>> >>>>>>>>>>>> #define VIDIOC_QUERY_EXT_CTRL _IOWR('V', 103, struct >>>>>>>>>>>> v4l2_query_ext_ctrl) >>>>>>>>>>>> +#define VIDIOC_DELETE_BUF _IOWR('V', 104, struct >>>>>>>>>>>> v4l2_buffer) >>>>>>>>>>>> + >>>>>>>>>>>> >>>>>>>>>>>> /* Reminder: when adding new ioctls please add support for >>>>>>>>>>>> them to >>>>>>>>>>>> drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */ >>>>>>>>>>>> -- >>>>>>>>>>>> 2.39.2 >>>>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Hsia-Jun(Randy) Li >>>>>>>>> >> -- >> Hsia-Jun(Randy) Li ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v3 10/11] media: v4l2: Add DELETE_BUF ioctl 2023-06-30 9:43 ` [PATCH v3 10/11] media: v4l2: Add DELETE_BUF ioctl Hsia-Jun Li 2023-07-03 8:12 ` Benjamin Gaignard @ 2023-07-13 9:09 ` Tomasz Figa 2023-07-17 2:16 ` Hsia-Jun Li 1 sibling, 1 reply; 27+ messages in thread From: Tomasz Figa @ 2023-07-13 9:09 UTC (permalink / raw) To: Hsia-Jun Li Cc: Benjamin Gaignard, linux-media, linux-kernel, p.zabel, ming.qian, hverkuil-cisco, gregkh, ezequiel, linux-arm-kernel, linux-mediatek, linux-arm-msm, linux-rockchip, nicolas.dufresne, linux-staging, kernel, mchehab, m.szyprowski, ayaka On Fri, Jun 30, 2023 at 05:43:51PM +0800, Hsia-Jun Li wrote: > > On 6/27/23 16:47, Hsia-Jun Li wrote: > > CAUTION: Email originated externally, do not click links or open > > attachments unless you recognize the sender and know the content is > > safe. > > > > > > On 6/27/23 16:43, Benjamin Gaignard wrote: > > > CAUTION: Email originated externally, do not click links or open > > > attachments unless you recognize the sender and know the content is > > > safe. > > > > > > > > > Le 27/06/2023 à 09:30, Hsia-Jun Li a écrit : > > > > > > > > On 6/22/23 21:13, Benjamin Gaignard wrote: > > > > > CAUTION: Email originated externally, do not click links or open > > > > > attachments unless you recognize the sender and know the content is > > > > > safe. > > > > > > > > > > > > > > > VIDIOC_DELETE_BUF ioctl allows to delete a buffer from a queue. > > > > > > > > > > Signed-off-by: Benjamin Gaignard <benjamin.gaignard@collabora.com> > > > > > --- > > > > > .../userspace-api/media/v4l/user-func.rst | 1 + > > > > > .../media/v4l/vidioc-delete-buf.rst | 51 > > > > > +++++++++++++++++++ > > > > > .../media/common/videobuf2/videobuf2-core.c | 33 ++++++++++++ > > > > > .../media/common/videobuf2/videobuf2-v4l2.c | 6 +++ > > > > > drivers/media/v4l2-core/v4l2-dev.c | 1 + > > > > > drivers/media/v4l2-core/v4l2-ioctl.c | 10 ++++ > > > > > include/media/v4l2-ioctl.h | 4 ++ > > > > > include/media/videobuf2-core.h | 9 ++++ > > > > > include/media/videobuf2-v4l2.h | 11 ++++ > > > > > include/uapi/linux/videodev2.h | 2 + > > > > > 10 files changed, 128 insertions(+) > > > > > create mode 100644 > > > > > Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst > > > > > > > > > > diff --git a/Documentation/userspace-api/media/v4l/user-func.rst > > > > > b/Documentation/userspace-api/media/v4l/user-func.rst > > > > > index 15ff0bf7bbe6..8c74016e12fd 100644 > > > > > --- a/Documentation/userspace-api/media/v4l/user-func.rst > > > > > +++ b/Documentation/userspace-api/media/v4l/user-func.rst > > > > > @@ -17,6 +17,7 @@ Function Reference > > > > > vidioc-dbg-g-chip-info > > > > > vidioc-dbg-g-register > > > > > vidioc-decoder-cmd > > > > > + vidioc-delete-buf > > > > > vidioc-dqevent > > > > > vidioc-dv-timings-cap > > > > > vidioc-encoder-cmd > > > > > diff --git > > > > > a/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst > > > > > b/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst > > > > > new file mode 100644 > > > > > index 000000000000..0e7ce58f91bc > > > > > --- /dev/null > > > > > +++ b/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst > > > > > @@ -0,0 +1,51 @@ > > > > > +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later > > > > > +.. c:namespace:: V4L > > > > > + > > > > > +.. _VIDIOC_DELETE_BUF: > > > > > + > > > > > +************************ > > > > > +ioctl VIDIOC_DELETE_BUF > > > > > +************************ > > > > > + > > > > > +Name > > > > > +==== > > > > > + > > > > > +VIDIOC_DELETE_BUF - Delete a buffer from a queue > > > > > + > > > > > +Synopsis > > > > > +======== > > > > > + > > > > > +.. c:macro:: VIDIOC_DELETE_BUF > > > > > + > > > > > +``int ioctl(int fd, VIDIOC_DELETE_BUF, struct v4l2_buffer *argp)`` > > > > > + > > > > > +Arguments > > > > > +========= > > > > > + > > > > > +``fd`` > > > > > + File descriptor returned by :c:func:`open()`. > > > > > + > > > > > +``argp`` > > > > > + Pointer to struct :c:type:`v4l2_buffer`. > > > > > + > > > > > +Description > > > > > +=========== > > > > > + > > > > > +Applications can optionally call the > > > > > :ref:`VIDIOC_DELETE_BUF` ioctl to > > > > > +delete a buffer from a queue. > > > > > + > > > > > +The struct :c:type:`v4l2_buffer` structure is specified in > > > > > +:ref:`buffer`. > > > > > + > > > > > +Return Value > > > > > +============ > > > > > + > > > > > +On success 0 is returned, on error -1 and the ``errno`` variable is > > > > > set > > > > > +appropriately. The generic error codes are described at the > > > > > +:ref:`Generic Error Codes <gen-errors>` chapter. > > > > > + > > > > > +EBUSY > > > > > + File I/O is in progress. > > > > > + > > > > > +EINVAL > > > > > + The buffer ``index`` doesn't exist in the queue. > > > > > diff --git a/drivers/media/common/videobuf2/videobuf2-core.c > > > > > b/drivers/media/common/videobuf2/videobuf2-core.c > > > > > index 899783f67580..aa546c972c3d 100644 > > > > > --- a/drivers/media/common/videobuf2/videobuf2-core.c > > > > > +++ b/drivers/media/common/videobuf2/videobuf2-core.c > > > > > @@ -1637,6 +1637,39 @@ int vb2_core_prepare_buf(struct vb2_queue *q, > > > > > unsigned int index, void *pb) > > > > > } > > > > > EXPORT_SYMBOL_GPL(vb2_core_prepare_buf); > > > > > > > > > > +int vb2_core_delete_buf(struct vb2_queue *q, unsigned int index) > > > > > +{ > > > > > + struct vb2_buffer *vb; > > > > > + > > > > > + vb = vb2_get_buffer(q, index); > > > > > + if (!vb) { > > > > > + dprintk(q, 1, "invalid buffer index %d\n", index); > > > > > + return -EINVAL; > > > > > + } > > > > > + > > > > > + if (vb->state != VB2_BUF_STATE_DEQUEUED) { > > > > > + dprintk(q, 1, "can't delete non dequeued buffer index > > > > > %d\n", index); > > > > > + return -EINVAL; > > > > > + } > > > > > + > > > > I know the driver could implement its own > > > > v4l2_ioctl_ops->vidioc_delete_buf() that check whether a buffer is > > > > used by the hardware as a future reference frame. > > > > But I think we need a flag to let the user know which buffer is still > > > > used by the hardware. > > > > Alternative ref case is safe, we only know it's existing when it is > > > > dequeued in current V4L2 buffer mechanism. > > > > While the Golden reference frame, such long term reference frame could > > > > last much longer. > > > > > > It is up to userland stack to know frames life time, it got the > > > information for that. > > > > That is true for the stateless codec driver. > > > > While application for stateful decoder could never do that. It also > > breaks what the document said: > > > > "The backing memory of |CAPTURE| buffers that are used as reference > > frames by the stream may be read by the hardware even after they are > > dequeued. Consequently, the client should avoid writing into this memory > > while the |CAPTURE| queue is streaming. Failure to observe this may > > result in corruption of decoded frames." > > > > > > > > > > + if (vb->planes[0].mem_priv) > > > > > + call_void_vb_qop(vb, buf_cleanup, vb); > > > > > + > > > > > + /* Free MMAP buffers or release USERPTR buffers */ > > > > > + if (q->memory == VB2_MEMORY_MMAP) > > > > > + __vb2_buf_mem_free(vb); > > Here is another problem for the existing application, the mmap() from the > mmap offset or exportbuffer fd would not create a reference to buffer in > this step(while the exportbuffer would create one itself). > > When you delete a buffer, you may not release it from its virtual memory > space, leaving a corrupted virtual memory space. What do you mean? __vb2_buf_mem_free() doesn't unconditionally free the memory, it just decrements a reference counter. The VMA holds its own, so the buffer is only fully released when the application calls munmap(). Best regards, Tomasz > Also this behavior is > right, because mmap(2) says: > > "After the mmap() call has returned, the file descriptor, fd, can be > closed immediately without invalidating the map‐ping." > > > > > > + else if (q->memory == VB2_MEMORY_DMABUF) > > > > > + __vb2_buf_dmabuf_put(vb); > > > > > + else > > > > > + __vb2_buf_userptr_put(vb); > > > > > + > > > > > + vb2_queue_remove_buffer(q, vb); > > > > > + kfree(vb); > > > > > + > > > > > + dprintk(q, 2, "buffer %d deleted\n", index); > > > > > + return 0; > > > > > +} > > > > > + > > > > > /* > > > > > * vb2_start_streaming() - Attempt to start streaming. > > > > > * @q: videobuf2 queue > > > > > diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c > > > > > b/drivers/media/common/videobuf2/videobuf2-v4l2.c > > > > > index 724135d41f7f..cea666c17b41 100644 > > > > > --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c > > > > > +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c > > > > > @@ -751,6 +751,12 @@ int vb2_prepare_buf(struct vb2_queue *q, struct > > > > > media_device *mdev, > > > > > } > > > > > EXPORT_SYMBOL_GPL(vb2_prepare_buf); > > > > > > > > > > +int vb2_delete_buf(struct vb2_queue *q, struct v4l2_buffer *b) > > > > > +{ > > > > > + return vb2_core_delete_buf(q, b->index); > > > > > +} > > > > > +EXPORT_SYMBOL_GPL(vb2_delete_buf); > > > > > + > > > > > int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers > > > > > *create) > > > > > { > > > > > unsigned requested_planes = 1; > > > > > diff --git a/drivers/media/v4l2-core/v4l2-dev.c > > > > > b/drivers/media/v4l2-core/v4l2-dev.c > > > > > index f81279492682..80ace2e1e932 100644 > > > > > --- a/drivers/media/v4l2-core/v4l2-dev.c > > > > > +++ b/drivers/media/v4l2-core/v4l2-dev.c > > > > > @@ -720,6 +720,7 @@ static void determine_valid_ioctls(struct > > > > > video_device *vdev) > > > > > SET_VALID_IOCTL(ops, VIDIOC_PREPARE_BUF, > > > > > vidioc_prepare_buf); > > > > > SET_VALID_IOCTL(ops, VIDIOC_STREAMON, > > > > > vidioc_streamon); > > > > > SET_VALID_IOCTL(ops, VIDIOC_STREAMOFF, > > > > > vidioc_streamoff); > > > > > + SET_VALID_IOCTL(ops, VIDIOC_DELETE_BUF, > > > > > vidioc_delete_buf); > > > > > } > > > > > > > > > > if (is_vid || is_vbi || is_meta) { > > > > > diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c > > > > > b/drivers/media/v4l2-core/v4l2-ioctl.c > > > > > index a858acea6547..1c737279d3ef 100644 > > > > > --- a/drivers/media/v4l2-core/v4l2-ioctl.c > > > > > +++ b/drivers/media/v4l2-core/v4l2-ioctl.c > > > > > @@ -2156,6 +2156,15 @@ static int v4l_prepare_buf(const struct > > > > > v4l2_ioctl_ops *ops, > > > > > return ret ? ret : ops->vidioc_prepare_buf(file, fh, b); > > > > > } > > > > > > > > > > +static int v4l_delete_buf(const struct v4l2_ioctl_ops *ops, > > > > > + struct file *file, void *fh, void *arg) > > > > > +{ > > > > > + struct v4l2_buffer *b = arg; > > > > > + int ret = check_fmt(file, b->type); > > > > > + > > > > > + return ret ? ret : ops->vidioc_delete_buf(file, fh, b); > > > > > +} > > > > > + > > > > > static int v4l_g_parm(const struct v4l2_ioctl_ops *ops, > > > > > struct file *file, void *fh, void > > > > > *arg) > > > > > { > > > > > @@ -2905,6 +2914,7 @@ static const struct v4l2_ioctl_info > > > > > v4l2_ioctls[] = { > > > > > IOCTL_INFO(VIDIOC_ENUM_FREQ_BANDS, v4l_enum_freq_bands, > > > > > v4l_print_freq_band, 0), > > > > > IOCTL_INFO(VIDIOC_DBG_G_CHIP_INFO, v4l_dbg_g_chip_info, > > > > > v4l_print_dbg_chip_info, INFO_FL_CLEAR(v4l2_dbg_chip_info, match)), > > > > > IOCTL_INFO(VIDIOC_QUERY_EXT_CTRL, v4l_query_ext_ctrl, > > > > > v4l_print_query_ext_ctrl, INFO_FL_CTRL | > > > > > INFO_FL_CLEAR(v4l2_query_ext_ctrl, id)), > > > > > + IOCTL_INFO(VIDIOC_DELETE_BUF, v4l_delete_buf, > > > > > v4l_print_buffer, INFO_FL_QUEUE), > > > > > }; > > > > > #define V4L2_IOCTLS ARRAY_SIZE(v4l2_ioctls) > > > > > > > > > > diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h > > > > > index edb733f21604..2f232ed884c7 100644 > > > > > --- a/include/media/v4l2-ioctl.h > > > > > +++ b/include/media/v4l2-ioctl.h > > > > > @@ -163,6 +163,8 @@ struct v4l2_fh; > > > > > * :ref:`VIDIOC_CREATE_BUFS <vidioc_create_bufs>` ioctl > > > > > * @vidioc_prepare_buf: pointer to the function that implements > > > > > * :ref:`VIDIOC_PREPARE_BUF <vidioc_prepare_buf>` ioctl > > > > > + * @vidioc_delete_buf: pointer to the function that implements > > > > > + * :ref:`VIDIOC_DELETE_BUF <vidioc_delete_buf>` ioctl > > > > > * @vidioc_overlay: pointer to the function that implements > > > > > * :ref:`VIDIOC_OVERLAY <vidioc_overlay>` ioctl > > > > > * @vidioc_g_fbuf: pointer to the function that implements > > > > > @@ -422,6 +424,8 @@ struct v4l2_ioctl_ops { > > > > > struct v4l2_create_buffers *b); > > > > > int (*vidioc_prepare_buf)(struct file *file, void *fh, > > > > > struct v4l2_buffer *b); > > > > > + int (*vidioc_delete_buf)(struct file *file, void *fh, > > > > > + struct v4l2_buffer *b); > > > > > > > > > > int (*vidioc_overlay)(struct file *file, void *fh, unsigned > > > > > int i); > > > > > int (*vidioc_g_fbuf)(struct file *file, void *fh, > > > > > diff --git a/include/media/videobuf2-core.h > > > > > b/include/media/videobuf2-core.h > > > > > index 080b783d608d..0f9e68f76b77 100644 > > > > > --- a/include/media/videobuf2-core.h > > > > > +++ b/include/media/videobuf2-core.h > > > > > @@ -840,6 +840,15 @@ int vb2_core_create_bufs(struct vb2_queue *q, > > > > > enum vb2_memory memory, > > > > > */ > > > > > int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, > > > > > void *pb); > > > > > > > > > > +/** > > > > > + * vb2_core_delete_buf() - > > > > > + * @q: pointer to &struct vb2_queue with videobuf2 queue. > > > > > + * @index: id number of the buffer. > > > > > + * > > > > > + * Return: returns zero on success; an error code otherwise. > > > > > + */ > > > > > +int vb2_core_delete_buf(struct vb2_queue *q, unsigned int index); > > > > > + > > > > > /** > > > > > * vb2_core_qbuf() - Queue a buffer from userspace > > > > > * > > > > > diff --git a/include/media/videobuf2-v4l2.h > > > > > b/include/media/videobuf2-v4l2.h > > > > > index 88a7a565170e..3beeb4c735f0 100644 > > > > > --- a/include/media/videobuf2-v4l2.h > > > > > +++ b/include/media/videobuf2-v4l2.h > > > > > @@ -114,6 +114,17 @@ int vb2_create_bufs(struct vb2_queue *q, struct > > > > > v4l2_create_buffers *create); > > > > > */ > > > > > int vb2_prepare_buf(struct vb2_queue *q, struct media_device *mdev, > > > > > struct v4l2_buffer *b); > > > > > +/** > > > > > + * vb2_delete_buf() - Delete the buffer from the queue > > > > > + * > > > > > + * @q: pointer to &struct vb2_queue with videobuf2 queue. > > > > > + * @b: buffer structure passed from userspace to > > > > > + * &v4l2_ioctl_ops->vidioc_delete_buf handler in driver > > > > > + * > > > > > + * The return values from this function are intended to be directly > > > > > returned > > > > > + * from &v4l2_ioctl_ops->vidioc_delete_buf handler in driver. > > > > > + */ > > > > > +int vb2_delete_buf(struct vb2_queue *q, struct v4l2_buffer *b); > > > > > > > > > > /** > > > > > * vb2_qbuf() - Queue a buffer from userspace > > > > > diff --git a/include/uapi/linux/videodev2.h > > > > > b/include/uapi/linux/videodev2.h > > > > > index aee75eb9e686..31bba1915642 100644 > > > > > --- a/include/uapi/linux/videodev2.h > > > > > +++ b/include/uapi/linux/videodev2.h > > > > > @@ -2702,6 +2702,8 @@ struct v4l2_create_buffers { > > > > > #define VIDIOC_DBG_G_CHIP_INFO _IOWR('V', 102, struct > > > > > v4l2_dbg_chip_info) > > > > > > > > > > #define VIDIOC_QUERY_EXT_CTRL _IOWR('V', 103, struct > > > > > v4l2_query_ext_ctrl) > > > > > +#define VIDIOC_DELETE_BUF _IOWR('V', 104, struct v4l2_buffer) > > > > > + > > > > > > > > > > /* Reminder: when adding new ioctls please add support for them to > > > > > drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */ > > > > > -- > > > > > 2.39.2 > > > > > > > -- > > Hsia-Jun(Randy) Li > > > -- > Hsia-Jun(Randy) Li > ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v3 10/11] media: v4l2: Add DELETE_BUF ioctl 2023-07-13 9:09 ` Tomasz Figa @ 2023-07-17 2:16 ` Hsia-Jun Li 2023-07-28 6:57 ` Tomasz Figa 0 siblings, 1 reply; 27+ messages in thread From: Hsia-Jun Li @ 2023-07-17 2:16 UTC (permalink / raw) To: Tomasz Figa Cc: Benjamin Gaignard, linux-media, linux-kernel, p.zabel, ming.qian, hverkuil-cisco, gregkh, ezequiel, linux-arm-kernel, linux-mediatek, linux-arm-msm, linux-rockchip, nicolas.dufresne, linux-staging, kernel, mchehab, m.szyprowski, ayaka On 7/13/23 17:09, Tomasz Figa wrote: > CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe. > > > On Fri, Jun 30, 2023 at 05:43:51PM +0800, Hsia-Jun Li wrote: >> On 6/27/23 16:47, Hsia-Jun Li wrote: >>> CAUTION: Email originated externally, do not click links or open >>> attachments unless you recognize the sender and know the content is >>> safe. >>> >>> >>> On 6/27/23 16:43, Benjamin Gaignard wrote: >>>> CAUTION: Email originated externally, do not click links or open >>>> attachments unless you recognize the sender and know the content is >>>> safe. >>>> >>>> >>>> Le 27/06/2023 à 09:30, Hsia-Jun Li a écrit : >>>>> On 6/22/23 21:13, Benjamin Gaignard wrote: >>>>>> CAUTION: Email originated externally, do not click links or open >>>>>> attachments unless you recognize the sender and know the content is >>>>>> safe. >>>>>> >>>>>> >>>>>> VIDIOC_DELETE_BUF ioctl allows to delete a buffer from a queue. >>>>>> >>>>>> Signed-off-by: Benjamin Gaignard <benjamin.gaignard@collabora.com> >>>>>> --- >>>>>> .../userspace-api/media/v4l/user-func.rst | 1 + >>>>>> .../media/v4l/vidioc-delete-buf.rst | 51 >>>>>> +++++++++++++++++++ >>>>>> .../media/common/videobuf2/videobuf2-core.c | 33 ++++++++++++ >>>>>> .../media/common/videobuf2/videobuf2-v4l2.c | 6 +++ >>>>>> drivers/media/v4l2-core/v4l2-dev.c | 1 + >>>>>> drivers/media/v4l2-core/v4l2-ioctl.c | 10 ++++ >>>>>> include/media/v4l2-ioctl.h | 4 ++ >>>>>> include/media/videobuf2-core.h | 9 ++++ >>>>>> include/media/videobuf2-v4l2.h | 11 ++++ >>>>>> include/uapi/linux/videodev2.h | 2 + >>>>>> 10 files changed, 128 insertions(+) >>>>>> create mode 100644 >>>>>> Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>> >>>>>> diff --git a/Documentation/userspace-api/media/v4l/user-func.rst >>>>>> b/Documentation/userspace-api/media/v4l/user-func.rst >>>>>> index 15ff0bf7bbe6..8c74016e12fd 100644 >>>>>> --- a/Documentation/userspace-api/media/v4l/user-func.rst >>>>>> +++ b/Documentation/userspace-api/media/v4l/user-func.rst >>>>>> @@ -17,6 +17,7 @@ Function Reference >>>>>> vidioc-dbg-g-chip-info >>>>>> vidioc-dbg-g-register >>>>>> vidioc-decoder-cmd >>>>>> + vidioc-delete-buf >>>>>> vidioc-dqevent >>>>>> vidioc-dv-timings-cap >>>>>> vidioc-encoder-cmd >>>>>> diff --git >>>>>> a/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>> b/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>> new file mode 100644 >>>>>> index 000000000000..0e7ce58f91bc >>>>>> --- /dev/null >>>>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>> @@ -0,0 +1,51 @@ >>>>>> +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later >>>>>> +.. c:namespace:: V4L >>>>>> + >>>>>> +.. _VIDIOC_DELETE_BUF: >>>>>> + >>>>>> +************************ >>>>>> +ioctl VIDIOC_DELETE_BUF >>>>>> +************************ >>>>>> + >>>>>> +Name >>>>>> +==== >>>>>> + >>>>>> +VIDIOC_DELETE_BUF - Delete a buffer from a queue >>>>>> + >>>>>> +Synopsis >>>>>> +======== >>>>>> + >>>>>> +.. c:macro:: VIDIOC_DELETE_BUF >>>>>> + >>>>>> +``int ioctl(int fd, VIDIOC_DELETE_BUF, struct v4l2_buffer *argp)`` >>>>>> + >>>>>> +Arguments >>>>>> +========= >>>>>> + >>>>>> +``fd`` >>>>>> + File descriptor returned by :c:func:`open()`. >>>>>> + >>>>>> +``argp`` >>>>>> + Pointer to struct :c:type:`v4l2_buffer`. >>>>>> + >>>>>> +Description >>>>>> +=========== >>>>>> + >>>>>> +Applications can optionally call the >>>>>> :ref:`VIDIOC_DELETE_BUF` ioctl to >>>>>> +delete a buffer from a queue. >>>>>> + >>>>>> +The struct :c:type:`v4l2_buffer` structure is specified in >>>>>> +:ref:`buffer`. >>>>>> + >>>>>> +Return Value >>>>>> +============ >>>>>> + >>>>>> +On success 0 is returned, on error -1 and the ``errno`` variable is >>>>>> set >>>>>> +appropriately. The generic error codes are described at the >>>>>> +:ref:`Generic Error Codes <gen-errors>` chapter. >>>>>> + >>>>>> +EBUSY >>>>>> + File I/O is in progress. >>>>>> + >>>>>> +EINVAL >>>>>> + The buffer ``index`` doesn't exist in the queue. >>>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c >>>>>> b/drivers/media/common/videobuf2/videobuf2-core.c >>>>>> index 899783f67580..aa546c972c3d 100644 >>>>>> --- a/drivers/media/common/videobuf2/videobuf2-core.c >>>>>> +++ b/drivers/media/common/videobuf2/videobuf2-core.c >>>>>> @@ -1637,6 +1637,39 @@ int vb2_core_prepare_buf(struct vb2_queue *q, >>>>>> unsigned int index, void *pb) >>>>>> } >>>>>> EXPORT_SYMBOL_GPL(vb2_core_prepare_buf); >>>>>> >>>>>> +int vb2_core_delete_buf(struct vb2_queue *q, unsigned int index) >>>>>> +{ >>>>>> + struct vb2_buffer *vb; >>>>>> + >>>>>> + vb = vb2_get_buffer(q, index); >>>>>> + if (!vb) { >>>>>> + dprintk(q, 1, "invalid buffer index %d\n", index); >>>>>> + return -EINVAL; >>>>>> + } >>>>>> + >>>>>> + if (vb->state != VB2_BUF_STATE_DEQUEUED) { >>>>>> + dprintk(q, 1, "can't delete non dequeued buffer index >>>>>> %d\n", index); >>>>>> + return -EINVAL; >>>>>> + } >>>>>> + >>>>> I know the driver could implement its own >>>>> v4l2_ioctl_ops->vidioc_delete_buf() that check whether a buffer is >>>>> used by the hardware as a future reference frame. >>>>> But I think we need a flag to let the user know which buffer is still >>>>> used by the hardware. >>>>> Alternative ref case is safe, we only know it's existing when it is >>>>> dequeued in current V4L2 buffer mechanism. >>>>> While the Golden reference frame, such long term reference frame could >>>>> last much longer. >>>> It is up to userland stack to know frames life time, it got the >>>> information for that. >>> That is true for the stateless codec driver. >>> >>> While application for stateful decoder could never do that. It also >>> breaks what the document said: >>> >>> "The backing memory of |CAPTURE| buffers that are used as reference >>> frames by the stream may be read by the hardware even after they are >>> dequeued. Consequently, the client should avoid writing into this memory >>> while the |CAPTURE| queue is streaming. Failure to observe this may >>> result in corruption of decoded frames." >>> >>>>>> + if (vb->planes[0].mem_priv) >>>>>> + call_void_vb_qop(vb, buf_cleanup, vb); >>>>>> + >>>>>> + /* Free MMAP buffers or release USERPTR buffers */ >>>>>> + if (q->memory == VB2_MEMORY_MMAP) >>>>>> + __vb2_buf_mem_free(vb); >> Here is another problem for the existing application, the mmap() from the >> mmap offset or exportbuffer fd would not create a reference to buffer in >> this step(while the exportbuffer would create one itself). >> >> When you delete a buffer, you may not release it from its virtual memory >> space, leaving a corrupted virtual memory space. > What do you mean? __vb2_buf_mem_free() doesn't unconditionally free the > memory, it just decrements a reference counter. struct dma_buf_ops->mmap() may not increase a reference to its buffer. While struct vb2_mem_ops->get_dmabuf() would. > The VMA holds its own, > so the buffer is only fully released when the application calls > munmap(). DELETE_BUF ioctl() didn't answer to this problem. Should the DELETE_BUF ioctl() make the no other user could access to this. > > Best regards, > Tomasz > >> Also this behavior is >> right, because mmap(2) says: >> >> "After the mmap() call has returned, the file descriptor, fd, can be >> closed immediately without invalidating the map‐ping." >> >>>>>> + else if (q->memory == VB2_MEMORY_DMABUF) >>>>>> + __vb2_buf_dmabuf_put(vb); >>>>>> + else >>>>>> + __vb2_buf_userptr_put(vb); >>>>>> + >>>>>> + vb2_queue_remove_buffer(q, vb); >>>>>> + kfree(vb); >>>>>> + >>>>>> + dprintk(q, 2, "buffer %d deleted\n", index); >>>>>> + return 0; >>>>>> +} >>>>>> + >>>>>> /* >>>>>> * vb2_start_streaming() - Attempt to start streaming. >>>>>> * @q: videobuf2 queue >>>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>> b/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>> index 724135d41f7f..cea666c17b41 100644 >>>>>> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>> @@ -751,6 +751,12 @@ int vb2_prepare_buf(struct vb2_queue *q, struct >>>>>> media_device *mdev, >>>>>> } >>>>>> EXPORT_SYMBOL_GPL(vb2_prepare_buf); >>>>>> >>>>>> +int vb2_delete_buf(struct vb2_queue *q, struct v4l2_buffer *b) >>>>>> +{ >>>>>> + return vb2_core_delete_buf(q, b->index); >>>>>> +} >>>>>> +EXPORT_SYMBOL_GPL(vb2_delete_buf); >>>>>> + >>>>>> int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers >>>>>> *create) >>>>>> { >>>>>> unsigned requested_planes = 1; >>>>>> diff --git a/drivers/media/v4l2-core/v4l2-dev.c >>>>>> b/drivers/media/v4l2-core/v4l2-dev.c >>>>>> index f81279492682..80ace2e1e932 100644 >>>>>> --- a/drivers/media/v4l2-core/v4l2-dev.c >>>>>> +++ b/drivers/media/v4l2-core/v4l2-dev.c >>>>>> @@ -720,6 +720,7 @@ static void determine_valid_ioctls(struct >>>>>> video_device *vdev) >>>>>> SET_VALID_IOCTL(ops, VIDIOC_PREPARE_BUF, >>>>>> vidioc_prepare_buf); >>>>>> SET_VALID_IOCTL(ops, VIDIOC_STREAMON, >>>>>> vidioc_streamon); >>>>>> SET_VALID_IOCTL(ops, VIDIOC_STREAMOFF, >>>>>> vidioc_streamoff); >>>>>> + SET_VALID_IOCTL(ops, VIDIOC_DELETE_BUF, >>>>>> vidioc_delete_buf); >>>>>> } >>>>>> >>>>>> if (is_vid || is_vbi || is_meta) { >>>>>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>> b/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>> index a858acea6547..1c737279d3ef 100644 >>>>>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>> @@ -2156,6 +2156,15 @@ static int v4l_prepare_buf(const struct >>>>>> v4l2_ioctl_ops *ops, >>>>>> return ret ? ret : ops->vidioc_prepare_buf(file, fh, b); >>>>>> } >>>>>> >>>>>> +static int v4l_delete_buf(const struct v4l2_ioctl_ops *ops, >>>>>> + struct file *file, void *fh, void *arg) >>>>>> +{ >>>>>> + struct v4l2_buffer *b = arg; >>>>>> + int ret = check_fmt(file, b->type); >>>>>> + >>>>>> + return ret ? ret : ops->vidioc_delete_buf(file, fh, b); >>>>>> +} >>>>>> + >>>>>> static int v4l_g_parm(const struct v4l2_ioctl_ops *ops, >>>>>> struct file *file, void *fh, void >>>>>> *arg) >>>>>> { >>>>>> @@ -2905,6 +2914,7 @@ static const struct v4l2_ioctl_info >>>>>> v4l2_ioctls[] = { >>>>>> IOCTL_INFO(VIDIOC_ENUM_FREQ_BANDS, v4l_enum_freq_bands, >>>>>> v4l_print_freq_band, 0), >>>>>> IOCTL_INFO(VIDIOC_DBG_G_CHIP_INFO, v4l_dbg_g_chip_info, >>>>>> v4l_print_dbg_chip_info, INFO_FL_CLEAR(v4l2_dbg_chip_info, match)), >>>>>> IOCTL_INFO(VIDIOC_QUERY_EXT_CTRL, v4l_query_ext_ctrl, >>>>>> v4l_print_query_ext_ctrl, INFO_FL_CTRL | >>>>>> INFO_FL_CLEAR(v4l2_query_ext_ctrl, id)), >>>>>> + IOCTL_INFO(VIDIOC_DELETE_BUF, v4l_delete_buf, >>>>>> v4l_print_buffer, INFO_FL_QUEUE), >>>>>> }; >>>>>> #define V4L2_IOCTLS ARRAY_SIZE(v4l2_ioctls) >>>>>> >>>>>> diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h >>>>>> index edb733f21604..2f232ed884c7 100644 >>>>>> --- a/include/media/v4l2-ioctl.h >>>>>> +++ b/include/media/v4l2-ioctl.h >>>>>> @@ -163,6 +163,8 @@ struct v4l2_fh; >>>>>> * :ref:`VIDIOC_CREATE_BUFS <vidioc_create_bufs>` ioctl >>>>>> * @vidioc_prepare_buf: pointer to the function that implements >>>>>> * :ref:`VIDIOC_PREPARE_BUF <vidioc_prepare_buf>` ioctl >>>>>> + * @vidioc_delete_buf: pointer to the function that implements >>>>>> + * :ref:`VIDIOC_DELETE_BUF <vidioc_delete_buf>` ioctl >>>>>> * @vidioc_overlay: pointer to the function that implements >>>>>> * :ref:`VIDIOC_OVERLAY <vidioc_overlay>` ioctl >>>>>> * @vidioc_g_fbuf: pointer to the function that implements >>>>>> @@ -422,6 +424,8 @@ struct v4l2_ioctl_ops { >>>>>> struct v4l2_create_buffers *b); >>>>>> int (*vidioc_prepare_buf)(struct file *file, void *fh, >>>>>> struct v4l2_buffer *b); >>>>>> + int (*vidioc_delete_buf)(struct file *file, void *fh, >>>>>> + struct v4l2_buffer *b); >>>>>> >>>>>> int (*vidioc_overlay)(struct file *file, void *fh, unsigned >>>>>> int i); >>>>>> int (*vidioc_g_fbuf)(struct file *file, void *fh, >>>>>> diff --git a/include/media/videobuf2-core.h >>>>>> b/include/media/videobuf2-core.h >>>>>> index 080b783d608d..0f9e68f76b77 100644 >>>>>> --- a/include/media/videobuf2-core.h >>>>>> +++ b/include/media/videobuf2-core.h >>>>>> @@ -840,6 +840,15 @@ int vb2_core_create_bufs(struct vb2_queue *q, >>>>>> enum vb2_memory memory, >>>>>> */ >>>>>> int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, >>>>>> void *pb); >>>>>> >>>>>> +/** >>>>>> + * vb2_core_delete_buf() - >>>>>> + * @q: pointer to &struct vb2_queue with videobuf2 queue. >>>>>> + * @index: id number of the buffer. >>>>>> + * >>>>>> + * Return: returns zero on success; an error code otherwise. >>>>>> + */ >>>>>> +int vb2_core_delete_buf(struct vb2_queue *q, unsigned int index); >>>>>> + >>>>>> /** >>>>>> * vb2_core_qbuf() - Queue a buffer from userspace >>>>>> * >>>>>> diff --git a/include/media/videobuf2-v4l2.h >>>>>> b/include/media/videobuf2-v4l2.h >>>>>> index 88a7a565170e..3beeb4c735f0 100644 >>>>>> --- a/include/media/videobuf2-v4l2.h >>>>>> +++ b/include/media/videobuf2-v4l2.h >>>>>> @@ -114,6 +114,17 @@ int vb2_create_bufs(struct vb2_queue *q, struct >>>>>> v4l2_create_buffers *create); >>>>>> */ >>>>>> int vb2_prepare_buf(struct vb2_queue *q, struct media_device *mdev, >>>>>> struct v4l2_buffer *b); >>>>>> +/** >>>>>> + * vb2_delete_buf() - Delete the buffer from the queue >>>>>> + * >>>>>> + * @q: pointer to &struct vb2_queue with videobuf2 queue. >>>>>> + * @b: buffer structure passed from userspace to >>>>>> + * &v4l2_ioctl_ops->vidioc_delete_buf handler in driver >>>>>> + * >>>>>> + * The return values from this function are intended to be directly >>>>>> returned >>>>>> + * from &v4l2_ioctl_ops->vidioc_delete_buf handler in driver. >>>>>> + */ >>>>>> +int vb2_delete_buf(struct vb2_queue *q, struct v4l2_buffer *b); >>>>>> >>>>>> /** >>>>>> * vb2_qbuf() - Queue a buffer from userspace >>>>>> diff --git a/include/uapi/linux/videodev2.h >>>>>> b/include/uapi/linux/videodev2.h >>>>>> index aee75eb9e686..31bba1915642 100644 >>>>>> --- a/include/uapi/linux/videodev2.h >>>>>> +++ b/include/uapi/linux/videodev2.h >>>>>> @@ -2702,6 +2702,8 @@ struct v4l2_create_buffers { >>>>>> #define VIDIOC_DBG_G_CHIP_INFO _IOWR('V', 102, struct >>>>>> v4l2_dbg_chip_info) >>>>>> >>>>>> #define VIDIOC_QUERY_EXT_CTRL _IOWR('V', 103, struct >>>>>> v4l2_query_ext_ctrl) >>>>>> +#define VIDIOC_DELETE_BUF _IOWR('V', 104, struct v4l2_buffer) >>>>>> + >>>>>> >>>>>> /* Reminder: when adding new ioctls please add support for them to >>>>>> drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */ >>>>>> -- >>>>>> 2.39.2 >>>>>> >>> -- >>> Hsia-Jun(Randy) Li >>> >> -- >> Hsia-Jun(Randy) Li >> -- Hsia-Jun(Randy) Li ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v3 10/11] media: v4l2: Add DELETE_BUF ioctl 2023-07-17 2:16 ` Hsia-Jun Li @ 2023-07-28 6:57 ` Tomasz Figa 2023-07-28 7:26 ` Hsia-Jun Li 0 siblings, 1 reply; 27+ messages in thread From: Tomasz Figa @ 2023-07-28 6:57 UTC (permalink / raw) To: Hsia-Jun Li Cc: Benjamin Gaignard, linux-media, linux-kernel, p.zabel, ming.qian, hverkuil-cisco, gregkh, ezequiel, linux-arm-kernel, linux-mediatek, linux-arm-msm, linux-rockchip, nicolas.dufresne, linux-staging, kernel, mchehab, m.szyprowski, ayaka On Mon, Jul 17, 2023 at 11:17 AM Hsia-Jun Li <Randy.Li@synaptics.com> wrote: > > > On 7/13/23 17:09, Tomasz Figa wrote: > > CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe. > > > > > > On Fri, Jun 30, 2023 at 05:43:51PM +0800, Hsia-Jun Li wrote: > >> On 6/27/23 16:47, Hsia-Jun Li wrote: > >>> CAUTION: Email originated externally, do not click links or open > >>> attachments unless you recognize the sender and know the content is > >>> safe. > >>> > >>> > >>> On 6/27/23 16:43, Benjamin Gaignard wrote: > >>>> CAUTION: Email originated externally, do not click links or open > >>>> attachments unless you recognize the sender and know the content is > >>>> safe. > >>>> > >>>> > >>>> Le 27/06/2023 à 09:30, Hsia-Jun Li a écrit : > >>>>> On 6/22/23 21:13, Benjamin Gaignard wrote: > >>>>>> CAUTION: Email originated externally, do not click links or open > >>>>>> attachments unless you recognize the sender and know the content is > >>>>>> safe. > >>>>>> > >>>>>> > >>>>>> VIDIOC_DELETE_BUF ioctl allows to delete a buffer from a queue. > >>>>>> > >>>>>> Signed-off-by: Benjamin Gaignard <benjamin.gaignard@collabora.com> > >>>>>> --- > >>>>>> .../userspace-api/media/v4l/user-func.rst | 1 + > >>>>>> .../media/v4l/vidioc-delete-buf.rst | 51 > >>>>>> +++++++++++++++++++ > >>>>>> .../media/common/videobuf2/videobuf2-core.c | 33 ++++++++++++ > >>>>>> .../media/common/videobuf2/videobuf2-v4l2.c | 6 +++ > >>>>>> drivers/media/v4l2-core/v4l2-dev.c | 1 + > >>>>>> drivers/media/v4l2-core/v4l2-ioctl.c | 10 ++++ > >>>>>> include/media/v4l2-ioctl.h | 4 ++ > >>>>>> include/media/videobuf2-core.h | 9 ++++ > >>>>>> include/media/videobuf2-v4l2.h | 11 ++++ > >>>>>> include/uapi/linux/videodev2.h | 2 + > >>>>>> 10 files changed, 128 insertions(+) > >>>>>> create mode 100644 > >>>>>> Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst > >>>>>> > >>>>>> diff --git a/Documentation/userspace-api/media/v4l/user-func.rst > >>>>>> b/Documentation/userspace-api/media/v4l/user-func.rst > >>>>>> index 15ff0bf7bbe6..8c74016e12fd 100644 > >>>>>> --- a/Documentation/userspace-api/media/v4l/user-func.rst > >>>>>> +++ b/Documentation/userspace-api/media/v4l/user-func.rst > >>>>>> @@ -17,6 +17,7 @@ Function Reference > >>>>>> vidioc-dbg-g-chip-info > >>>>>> vidioc-dbg-g-register > >>>>>> vidioc-decoder-cmd > >>>>>> + vidioc-delete-buf > >>>>>> vidioc-dqevent > >>>>>> vidioc-dv-timings-cap > >>>>>> vidioc-encoder-cmd > >>>>>> diff --git > >>>>>> a/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst > >>>>>> b/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst > >>>>>> new file mode 100644 > >>>>>> index 000000000000..0e7ce58f91bc > >>>>>> --- /dev/null > >>>>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst > >>>>>> @@ -0,0 +1,51 @@ > >>>>>> +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later > >>>>>> +.. c:namespace:: V4L > >>>>>> + > >>>>>> +.. _VIDIOC_DELETE_BUF: > >>>>>> + > >>>>>> +************************ > >>>>>> +ioctl VIDIOC_DELETE_BUF > >>>>>> +************************ > >>>>>> + > >>>>>> +Name > >>>>>> +==== > >>>>>> + > >>>>>> +VIDIOC_DELETE_BUF - Delete a buffer from a queue > >>>>>> + > >>>>>> +Synopsis > >>>>>> +======== > >>>>>> + > >>>>>> +.. c:macro:: VIDIOC_DELETE_BUF > >>>>>> + > >>>>>> +``int ioctl(int fd, VIDIOC_DELETE_BUF, struct v4l2_buffer *argp)`` > >>>>>> + > >>>>>> +Arguments > >>>>>> +========= > >>>>>> + > >>>>>> +``fd`` > >>>>>> + File descriptor returned by :c:func:`open()`. > >>>>>> + > >>>>>> +``argp`` > >>>>>> + Pointer to struct :c:type:`v4l2_buffer`. > >>>>>> + > >>>>>> +Description > >>>>>> +=========== > >>>>>> + > >>>>>> +Applications can optionally call the > >>>>>> :ref:`VIDIOC_DELETE_BUF` ioctl to > >>>>>> +delete a buffer from a queue. > >>>>>> + > >>>>>> +The struct :c:type:`v4l2_buffer` structure is specified in > >>>>>> +:ref:`buffer`. > >>>>>> + > >>>>>> +Return Value > >>>>>> +============ > >>>>>> + > >>>>>> +On success 0 is returned, on error -1 and the ``errno`` variable is > >>>>>> set > >>>>>> +appropriately. The generic error codes are described at the > >>>>>> +:ref:`Generic Error Codes <gen-errors>` chapter. > >>>>>> + > >>>>>> +EBUSY > >>>>>> + File I/O is in progress. > >>>>>> + > >>>>>> +EINVAL > >>>>>> + The buffer ``index`` doesn't exist in the queue. > >>>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c > >>>>>> b/drivers/media/common/videobuf2/videobuf2-core.c > >>>>>> index 899783f67580..aa546c972c3d 100644 > >>>>>> --- a/drivers/media/common/videobuf2/videobuf2-core.c > >>>>>> +++ b/drivers/media/common/videobuf2/videobuf2-core.c > >>>>>> @@ -1637,6 +1637,39 @@ int vb2_core_prepare_buf(struct vb2_queue *q, > >>>>>> unsigned int index, void *pb) > >>>>>> } > >>>>>> EXPORT_SYMBOL_GPL(vb2_core_prepare_buf); > >>>>>> > >>>>>> +int vb2_core_delete_buf(struct vb2_queue *q, unsigned int index) > >>>>>> +{ > >>>>>> + struct vb2_buffer *vb; > >>>>>> + > >>>>>> + vb = vb2_get_buffer(q, index); > >>>>>> + if (!vb) { > >>>>>> + dprintk(q, 1, "invalid buffer index %d\n", index); > >>>>>> + return -EINVAL; > >>>>>> + } > >>>>>> + > >>>>>> + if (vb->state != VB2_BUF_STATE_DEQUEUED) { > >>>>>> + dprintk(q, 1, "can't delete non dequeued buffer index > >>>>>> %d\n", index); > >>>>>> + return -EINVAL; > >>>>>> + } > >>>>>> + > >>>>> I know the driver could implement its own > >>>>> v4l2_ioctl_ops->vidioc_delete_buf() that check whether a buffer is > >>>>> used by the hardware as a future reference frame. > >>>>> But I think we need a flag to let the user know which buffer is still > >>>>> used by the hardware. > >>>>> Alternative ref case is safe, we only know it's existing when it is > >>>>> dequeued in current V4L2 buffer mechanism. > >>>>> While the Golden reference frame, such long term reference frame could > >>>>> last much longer. > >>>> It is up to userland stack to know frames life time, it got the > >>>> information for that. > >>> That is true for the stateless codec driver. > >>> > >>> While application for stateful decoder could never do that. It also > >>> breaks what the document said: > >>> > >>> "The backing memory of |CAPTURE| buffers that are used as reference > >>> frames by the stream may be read by the hardware even after they are > >>> dequeued. Consequently, the client should avoid writing into this memory > >>> while the |CAPTURE| queue is streaming. Failure to observe this may > >>> result in corruption of decoded frames." > >>> > >>>>>> + if (vb->planes[0].mem_priv) > >>>>>> + call_void_vb_qop(vb, buf_cleanup, vb); > >>>>>> + > >>>>>> + /* Free MMAP buffers or release USERPTR buffers */ > >>>>>> + if (q->memory == VB2_MEMORY_MMAP) > >>>>>> + __vb2_buf_mem_free(vb); > >> Here is another problem for the existing application, the mmap() from the > >> mmap offset or exportbuffer fd would not create a reference to buffer in > >> this step(while the exportbuffer would create one itself). > >> > >> When you delete a buffer, you may not release it from its virtual memory > >> space, leaving a corrupted virtual memory space. > > What do you mean? __vb2_buf_mem_free() doesn't unconditionally free the > > memory, it just decrements a reference counter. > > struct dma_buf_ops->mmap() may not increase a reference to its buffer. Both V4L2 mmap() and DMA-buf mmap() of buffers exported from V4L2 would increase a reference to the buffer. They both lead to vb2_{dc,sg,vmalloc}_mmap() which open the VMA, which in turn calls vb2_common_vm_open() that increases the buffer refcount. Best regards, Tomasz > > While struct vb2_mem_ops->get_dmabuf() would. > > > The VMA holds its own, > > so the buffer is only fully released when the application calls > > munmap(). > > DELETE_BUF ioctl() didn't answer to this problem. Should the DELETE_BUF > ioctl() make the no other user could access to this. > > > > > Best regards, > > Tomasz > > > >> Also this behavior is > >> right, because mmap(2) says: > >> > >> "After the mmap() call has returned, the file descriptor, fd, can be > >> closed immediately without invalidating the map‐ping." > >> > >>>>>> + else if (q->memory == VB2_MEMORY_DMABUF) > >>>>>> + __vb2_buf_dmabuf_put(vb); > >>>>>> + else > >>>>>> + __vb2_buf_userptr_put(vb); > >>>>>> + > >>>>>> + vb2_queue_remove_buffer(q, vb); > >>>>>> + kfree(vb); > >>>>>> + > >>>>>> + dprintk(q, 2, "buffer %d deleted\n", index); > >>>>>> + return 0; > >>>>>> +} > >>>>>> + > >>>>>> /* > >>>>>> * vb2_start_streaming() - Attempt to start streaming. > >>>>>> * @q: videobuf2 queue > >>>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c > >>>>>> b/drivers/media/common/videobuf2/videobuf2-v4l2.c > >>>>>> index 724135d41f7f..cea666c17b41 100644 > >>>>>> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c > >>>>>> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c > >>>>>> @@ -751,6 +751,12 @@ int vb2_prepare_buf(struct vb2_queue *q, struct > >>>>>> media_device *mdev, > >>>>>> } > >>>>>> EXPORT_SYMBOL_GPL(vb2_prepare_buf); > >>>>>> > >>>>>> +int vb2_delete_buf(struct vb2_queue *q, struct v4l2_buffer *b) > >>>>>> +{ > >>>>>> + return vb2_core_delete_buf(q, b->index); > >>>>>> +} > >>>>>> +EXPORT_SYMBOL_GPL(vb2_delete_buf); > >>>>>> + > >>>>>> int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers > >>>>>> *create) > >>>>>> { > >>>>>> unsigned requested_planes = 1; > >>>>>> diff --git a/drivers/media/v4l2-core/v4l2-dev.c > >>>>>> b/drivers/media/v4l2-core/v4l2-dev.c > >>>>>> index f81279492682..80ace2e1e932 100644 > >>>>>> --- a/drivers/media/v4l2-core/v4l2-dev.c > >>>>>> +++ b/drivers/media/v4l2-core/v4l2-dev.c > >>>>>> @@ -720,6 +720,7 @@ static void determine_valid_ioctls(struct > >>>>>> video_device *vdev) > >>>>>> SET_VALID_IOCTL(ops, VIDIOC_PREPARE_BUF, > >>>>>> vidioc_prepare_buf); > >>>>>> SET_VALID_IOCTL(ops, VIDIOC_STREAMON, > >>>>>> vidioc_streamon); > >>>>>> SET_VALID_IOCTL(ops, VIDIOC_STREAMOFF, > >>>>>> vidioc_streamoff); > >>>>>> + SET_VALID_IOCTL(ops, VIDIOC_DELETE_BUF, > >>>>>> vidioc_delete_buf); > >>>>>> } > >>>>>> > >>>>>> if (is_vid || is_vbi || is_meta) { > >>>>>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c > >>>>>> b/drivers/media/v4l2-core/v4l2-ioctl.c > >>>>>> index a858acea6547..1c737279d3ef 100644 > >>>>>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c > >>>>>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c > >>>>>> @@ -2156,6 +2156,15 @@ static int v4l_prepare_buf(const struct > >>>>>> v4l2_ioctl_ops *ops, > >>>>>> return ret ? ret : ops->vidioc_prepare_buf(file, fh, b); > >>>>>> } > >>>>>> > >>>>>> +static int v4l_delete_buf(const struct v4l2_ioctl_ops *ops, > >>>>>> + struct file *file, void *fh, void *arg) > >>>>>> +{ > >>>>>> + struct v4l2_buffer *b = arg; > >>>>>> + int ret = check_fmt(file, b->type); > >>>>>> + > >>>>>> + return ret ? ret : ops->vidioc_delete_buf(file, fh, b); > >>>>>> +} > >>>>>> + > >>>>>> static int v4l_g_parm(const struct v4l2_ioctl_ops *ops, > >>>>>> struct file *file, void *fh, void > >>>>>> *arg) > >>>>>> { > >>>>>> @@ -2905,6 +2914,7 @@ static const struct v4l2_ioctl_info > >>>>>> v4l2_ioctls[] = { > >>>>>> IOCTL_INFO(VIDIOC_ENUM_FREQ_BANDS, v4l_enum_freq_bands, > >>>>>> v4l_print_freq_band, 0), > >>>>>> IOCTL_INFO(VIDIOC_DBG_G_CHIP_INFO, v4l_dbg_g_chip_info, > >>>>>> v4l_print_dbg_chip_info, INFO_FL_CLEAR(v4l2_dbg_chip_info, match)), > >>>>>> IOCTL_INFO(VIDIOC_QUERY_EXT_CTRL, v4l_query_ext_ctrl, > >>>>>> v4l_print_query_ext_ctrl, INFO_FL_CTRL | > >>>>>> INFO_FL_CLEAR(v4l2_query_ext_ctrl, id)), > >>>>>> + IOCTL_INFO(VIDIOC_DELETE_BUF, v4l_delete_buf, > >>>>>> v4l_print_buffer, INFO_FL_QUEUE), > >>>>>> }; > >>>>>> #define V4L2_IOCTLS ARRAY_SIZE(v4l2_ioctls) > >>>>>> > >>>>>> diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h > >>>>>> index edb733f21604..2f232ed884c7 100644 > >>>>>> --- a/include/media/v4l2-ioctl.h > >>>>>> +++ b/include/media/v4l2-ioctl.h > >>>>>> @@ -163,6 +163,8 @@ struct v4l2_fh; > >>>>>> * :ref:`VIDIOC_CREATE_BUFS <vidioc_create_bufs>` ioctl > >>>>>> * @vidioc_prepare_buf: pointer to the function that implements > >>>>>> * :ref:`VIDIOC_PREPARE_BUF <vidioc_prepare_buf>` ioctl > >>>>>> + * @vidioc_delete_buf: pointer to the function that implements > >>>>>> + * :ref:`VIDIOC_DELETE_BUF <vidioc_delete_buf>` ioctl > >>>>>> * @vidioc_overlay: pointer to the function that implements > >>>>>> * :ref:`VIDIOC_OVERLAY <vidioc_overlay>` ioctl > >>>>>> * @vidioc_g_fbuf: pointer to the function that implements > >>>>>> @@ -422,6 +424,8 @@ struct v4l2_ioctl_ops { > >>>>>> struct v4l2_create_buffers *b); > >>>>>> int (*vidioc_prepare_buf)(struct file *file, void *fh, > >>>>>> struct v4l2_buffer *b); > >>>>>> + int (*vidioc_delete_buf)(struct file *file, void *fh, > >>>>>> + struct v4l2_buffer *b); > >>>>>> > >>>>>> int (*vidioc_overlay)(struct file *file, void *fh, unsigned > >>>>>> int i); > >>>>>> int (*vidioc_g_fbuf)(struct file *file, void *fh, > >>>>>> diff --git a/include/media/videobuf2-core.h > >>>>>> b/include/media/videobuf2-core.h > >>>>>> index 080b783d608d..0f9e68f76b77 100644 > >>>>>> --- a/include/media/videobuf2-core.h > >>>>>> +++ b/include/media/videobuf2-core.h > >>>>>> @@ -840,6 +840,15 @@ int vb2_core_create_bufs(struct vb2_queue *q, > >>>>>> enum vb2_memory memory, > >>>>>> */ > >>>>>> int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, > >>>>>> void *pb); > >>>>>> > >>>>>> +/** > >>>>>> + * vb2_core_delete_buf() - > >>>>>> + * @q: pointer to &struct vb2_queue with videobuf2 queue. > >>>>>> + * @index: id number of the buffer. > >>>>>> + * > >>>>>> + * Return: returns zero on success; an error code otherwise. > >>>>>> + */ > >>>>>> +int vb2_core_delete_buf(struct vb2_queue *q, unsigned int index); > >>>>>> + > >>>>>> /** > >>>>>> * vb2_core_qbuf() - Queue a buffer from userspace > >>>>>> * > >>>>>> diff --git a/include/media/videobuf2-v4l2.h > >>>>>> b/include/media/videobuf2-v4l2.h > >>>>>> index 88a7a565170e..3beeb4c735f0 100644 > >>>>>> --- a/include/media/videobuf2-v4l2.h > >>>>>> +++ b/include/media/videobuf2-v4l2.h > >>>>>> @@ -114,6 +114,17 @@ int vb2_create_bufs(struct vb2_queue *q, struct > >>>>>> v4l2_create_buffers *create); > >>>>>> */ > >>>>>> int vb2_prepare_buf(struct vb2_queue *q, struct media_device *mdev, > >>>>>> struct v4l2_buffer *b); > >>>>>> +/** > >>>>>> + * vb2_delete_buf() - Delete the buffer from the queue > >>>>>> + * > >>>>>> + * @q: pointer to &struct vb2_queue with videobuf2 queue. > >>>>>> + * @b: buffer structure passed from userspace to > >>>>>> + * &v4l2_ioctl_ops->vidioc_delete_buf handler in driver > >>>>>> + * > >>>>>> + * The return values from this function are intended to be directly > >>>>>> returned > >>>>>> + * from &v4l2_ioctl_ops->vidioc_delete_buf handler in driver. > >>>>>> + */ > >>>>>> +int vb2_delete_buf(struct vb2_queue *q, struct v4l2_buffer *b); > >>>>>> > >>>>>> /** > >>>>>> * vb2_qbuf() - Queue a buffer from userspace > >>>>>> diff --git a/include/uapi/linux/videodev2.h > >>>>>> b/include/uapi/linux/videodev2.h > >>>>>> index aee75eb9e686..31bba1915642 100644 > >>>>>> --- a/include/uapi/linux/videodev2.h > >>>>>> +++ b/include/uapi/linux/videodev2.h > >>>>>> @@ -2702,6 +2702,8 @@ struct v4l2_create_buffers { > >>>>>> #define VIDIOC_DBG_G_CHIP_INFO _IOWR('V', 102, struct > >>>>>> v4l2_dbg_chip_info) > >>>>>> > >>>>>> #define VIDIOC_QUERY_EXT_CTRL _IOWR('V', 103, struct > >>>>>> v4l2_query_ext_ctrl) > >>>>>> +#define VIDIOC_DELETE_BUF _IOWR('V', 104, struct v4l2_buffer) > >>>>>> + > >>>>>> > >>>>>> /* Reminder: when adding new ioctls please add support for them to > >>>>>> drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */ > >>>>>> -- > >>>>>> 2.39.2 > >>>>>> > >>> -- > >>> Hsia-Jun(Randy) Li > >>> > >> -- > >> Hsia-Jun(Randy) Li > >> > -- > Hsia-Jun(Randy) Li > ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v3 10/11] media: v4l2: Add DELETE_BUF ioctl 2023-07-28 6:57 ` Tomasz Figa @ 2023-07-28 7:26 ` Hsia-Jun Li 0 siblings, 0 replies; 27+ messages in thread From: Hsia-Jun Li @ 2023-07-28 7:26 UTC (permalink / raw) To: Tomasz Figa Cc: Benjamin Gaignard, linux-media, linux-kernel, p.zabel, ming.qian, hverkuil-cisco, gregkh, ezequiel, linux-arm-kernel, linux-mediatek, linux-arm-msm, linux-rockchip, nicolas.dufresne, linux-staging, kernel, mchehab, m.szyprowski, ayaka On 7/28/23 14:57, Tomasz Figa wrote: > CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe. > > > On Mon, Jul 17, 2023 at 11:17 AM Hsia-Jun Li <Randy.Li@synaptics.com> wrote: >> >> >> On 7/13/23 17:09, Tomasz Figa wrote: >>> CAUTION: Email originated externally, do not click links or open attachments unless you recognize the sender and know the content is safe. >>> >>> >>> On Fri, Jun 30, 2023 at 05:43:51PM +0800, Hsia-Jun Li wrote: >>>> On 6/27/23 16:47, Hsia-Jun Li wrote: >>>>> CAUTION: Email originated externally, do not click links or open >>>>> attachments unless you recognize the sender and know the content is >>>>> safe. >>>>> >>>>> >>>>> On 6/27/23 16:43, Benjamin Gaignard wrote: >>>>>> CAUTION: Email originated externally, do not click links or open >>>>>> attachments unless you recognize the sender and know the content is >>>>>> safe. >>>>>> >>>>>> >>>>>> Le 27/06/2023 à 09:30, Hsia-Jun Li a écrit : >>>>>>> On 6/22/23 21:13, Benjamin Gaignard wrote: >>>>>>>> CAUTION: Email originated externally, do not click links or open >>>>>>>> attachments unless you recognize the sender and know the content is >>>>>>>> safe. >>>>>>>> >>>>>>>> >>>>>>>> VIDIOC_DELETE_BUF ioctl allows to delete a buffer from a queue. >>>>>>>> >>>>>>>> Signed-off-by: Benjamin Gaignard <benjamin.gaignard@collabora.com> >>>>>>>> --- >>>>>>>> .../userspace-api/media/v4l/user-func.rst | 1 + >>>>>>>> .../media/v4l/vidioc-delete-buf.rst | 51 >>>>>>>> +++++++++++++++++++ >>>>>>>> .../media/common/videobuf2/videobuf2-core.c | 33 ++++++++++++ >>>>>>>> .../media/common/videobuf2/videobuf2-v4l2.c | 6 +++ >>>>>>>> drivers/media/v4l2-core/v4l2-dev.c | 1 + >>>>>>>> drivers/media/v4l2-core/v4l2-ioctl.c | 10 ++++ >>>>>>>> include/media/v4l2-ioctl.h | 4 ++ >>>>>>>> include/media/videobuf2-core.h | 9 ++++ >>>>>>>> include/media/videobuf2-v4l2.h | 11 ++++ >>>>>>>> include/uapi/linux/videodev2.h | 2 + >>>>>>>> 10 files changed, 128 insertions(+) >>>>>>>> create mode 100644 >>>>>>>> Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>>>> >>>>>>>> diff --git a/Documentation/userspace-api/media/v4l/user-func.rst >>>>>>>> b/Documentation/userspace-api/media/v4l/user-func.rst >>>>>>>> index 15ff0bf7bbe6..8c74016e12fd 100644 >>>>>>>> --- a/Documentation/userspace-api/media/v4l/user-func.rst >>>>>>>> +++ b/Documentation/userspace-api/media/v4l/user-func.rst >>>>>>>> @@ -17,6 +17,7 @@ Function Reference >>>>>>>> vidioc-dbg-g-chip-info >>>>>>>> vidioc-dbg-g-register >>>>>>>> vidioc-decoder-cmd >>>>>>>> + vidioc-delete-buf >>>>>>>> vidioc-dqevent >>>>>>>> vidioc-dv-timings-cap >>>>>>>> vidioc-encoder-cmd >>>>>>>> diff --git >>>>>>>> a/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>>>> b/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>>>> new file mode 100644 >>>>>>>> index 000000000000..0e7ce58f91bc >>>>>>>> --- /dev/null >>>>>>>> +++ b/Documentation/userspace-api/media/v4l/vidioc-delete-buf.rst >>>>>>>> @@ -0,0 +1,51 @@ >>>>>>>> +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later >>>>>>>> +.. c:namespace:: V4L >>>>>>>> + >>>>>>>> +.. _VIDIOC_DELETE_BUF: >>>>>>>> + >>>>>>>> +************************ >>>>>>>> +ioctl VIDIOC_DELETE_BUF >>>>>>>> +************************ >>>>>>>> + >>>>>>>> +Name >>>>>>>> +==== >>>>>>>> + >>>>>>>> +VIDIOC_DELETE_BUF - Delete a buffer from a queue >>>>>>>> + >>>>>>>> +Synopsis >>>>>>>> +======== >>>>>>>> + >>>>>>>> +.. c:macro:: VIDIOC_DELETE_BUF >>>>>>>> + >>>>>>>> +``int ioctl(int fd, VIDIOC_DELETE_BUF, struct v4l2_buffer *argp)`` >>>>>>>> + >>>>>>>> +Arguments >>>>>>>> +========= >>>>>>>> + >>>>>>>> +``fd`` >>>>>>>> + File descriptor returned by :c:func:`open()`. >>>>>>>> + >>>>>>>> +``argp`` >>>>>>>> + Pointer to struct :c:type:`v4l2_buffer`. >>>>>>>> + >>>>>>>> +Description >>>>>>>> +=========== >>>>>>>> + >>>>>>>> +Applications can optionally call the >>>>>>>> :ref:`VIDIOC_DELETE_BUF` ioctl to >>>>>>>> +delete a buffer from a queue. >>>>>>>> + >>>>>>>> +The struct :c:type:`v4l2_buffer` structure is specified in >>>>>>>> +:ref:`buffer`. >>>>>>>> + >>>>>>>> +Return Value >>>>>>>> +============ >>>>>>>> + >>>>>>>> +On success 0 is returned, on error -1 and the ``errno`` variable is >>>>>>>> set >>>>>>>> +appropriately. The generic error codes are described at the >>>>>>>> +:ref:`Generic Error Codes <gen-errors>` chapter. >>>>>>>> + >>>>>>>> +EBUSY >>>>>>>> + File I/O is in progress. >>>>>>>> + >>>>>>>> +EINVAL >>>>>>>> + The buffer ``index`` doesn't exist in the queue. >>>>>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>> b/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>> index 899783f67580..aa546c972c3d 100644 >>>>>>>> --- a/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>> +++ b/drivers/media/common/videobuf2/videobuf2-core.c >>>>>>>> @@ -1637,6 +1637,39 @@ int vb2_core_prepare_buf(struct vb2_queue *q, >>>>>>>> unsigned int index, void *pb) >>>>>>>> } >>>>>>>> EXPORT_SYMBOL_GPL(vb2_core_prepare_buf); >>>>>>>> >>>>>>>> +int vb2_core_delete_buf(struct vb2_queue *q, unsigned int index) >>>>>>>> +{ >>>>>>>> + struct vb2_buffer *vb; >>>>>>>> + >>>>>>>> + vb = vb2_get_buffer(q, index); >>>>>>>> + if (!vb) { >>>>>>>> + dprintk(q, 1, "invalid buffer index %d\n", index); >>>>>>>> + return -EINVAL; >>>>>>>> + } >>>>>>>> + >>>>>>>> + if (vb->state != VB2_BUF_STATE_DEQUEUED) { >>>>>>>> + dprintk(q, 1, "can't delete non dequeued buffer index >>>>>>>> %d\n", index); >>>>>>>> + return -EINVAL; >>>>>>>> + } >>>>>>>> + >>>>>>> I know the driver could implement its own >>>>>>> v4l2_ioctl_ops->vidioc_delete_buf() that check whether a buffer is >>>>>>> used by the hardware as a future reference frame. >>>>>>> But I think we need a flag to let the user know which buffer is still >>>>>>> used by the hardware. >>>>>>> Alternative ref case is safe, we only know it's existing when it is >>>>>>> dequeued in current V4L2 buffer mechanism. >>>>>>> While the Golden reference frame, such long term reference frame could >>>>>>> last much longer. >>>>>> It is up to userland stack to know frames life time, it got the >>>>>> information for that. >>>>> That is true for the stateless codec driver. >>>>> >>>>> While application for stateful decoder could never do that. It also >>>>> breaks what the document said: >>>>> >>>>> "The backing memory of |CAPTURE| buffers that are used as reference >>>>> frames by the stream may be read by the hardware even after they are >>>>> dequeued. Consequently, the client should avoid writing into this memory >>>>> while the |CAPTURE| queue is streaming. Failure to observe this may >>>>> result in corruption of decoded frames." >>>>> >>>>>>>> + if (vb->planes[0].mem_priv) >>>>>>>> + call_void_vb_qop(vb, buf_cleanup, vb); >>>>>>>> + >>>>>>>> + /* Free MMAP buffers or release USERPTR buffers */ >>>>>>>> + if (q->memory == VB2_MEMORY_MMAP) >>>>>>>> + __vb2_buf_mem_free(vb); >>>> Here is another problem for the existing application, the mmap() from the >>>> mmap offset or exportbuffer fd would not create a reference to buffer in >>>> this step(while the exportbuffer would create one itself). >>>> >>>> When you delete a buffer, you may not release it from its virtual memory >>>> space, leaving a corrupted virtual memory space. >>> What do you mean? __vb2_buf_mem_free() doesn't unconditionally free the >>> memory, it just decrements a reference counter. >> >> struct dma_buf_ops->mmap() may not increase a reference to its buffer. I think we are talking the same refcount. That is vb2_vmarea_handler->refcount. While, I am thinking about refcount from vb2_dc_buf. > > Both V4L2 mmap() and DMA-buf mmap() of buffers exported from V4L2 > would increase a reference to the buffer. They both lead to > vb2_{dc,sg,vmalloc}_mmap() which open the VMA, which in turn calls > vb2_common_vm_open() that increases the buffer refcount. > > Best regards, > Tomasz > >> >> While struct vb2_mem_ops->get_dmabuf() would. >> >>> The VMA holds its own, >>> so the buffer is only fully released when the application calls >>> munmap(). >> >> DELETE_BUF ioctl() didn't answer to this problem. Should the DELETE_BUF >> ioctl() make the no other user could access to this. >> >>> >>> Best regards, >>> Tomasz >>> >>>> Also this behavior is >>>> right, because mmap(2) says: >>>> >>>> "After the mmap() call has returned, the file descriptor, fd, can be >>>> closed immediately without invalidating the map‐ping." >>>> >>>>>>>> + else if (q->memory == VB2_MEMORY_DMABUF) >>>>>>>> + __vb2_buf_dmabuf_put(vb); >>>>>>>> + else >>>>>>>> + __vb2_buf_userptr_put(vb); >>>>>>>> + >>>>>>>> + vb2_queue_remove_buffer(q, vb); >>>>>>>> + kfree(vb); >>>>>>>> + >>>>>>>> + dprintk(q, 2, "buffer %d deleted\n", index); >>>>>>>> + return 0; >>>>>>>> +} >>>>>>>> + >>>>>>>> /* >>>>>>>> * vb2_start_streaming() - Attempt to start streaming. >>>>>>>> * @q: videobuf2 queue >>>>>>>> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>>>> b/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>>>> index 724135d41f7f..cea666c17b41 100644 >>>>>>>> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>>>> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c >>>>>>>> @@ -751,6 +751,12 @@ int vb2_prepare_buf(struct vb2_queue *q, struct >>>>>>>> media_device *mdev, >>>>>>>> } >>>>>>>> EXPORT_SYMBOL_GPL(vb2_prepare_buf); >>>>>>>> >>>>>>>> +int vb2_delete_buf(struct vb2_queue *q, struct v4l2_buffer *b) >>>>>>>> +{ >>>>>>>> + return vb2_core_delete_buf(q, b->index); >>>>>>>> +} >>>>>>>> +EXPORT_SYMBOL_GPL(vb2_delete_buf); >>>>>>>> + >>>>>>>> int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers >>>>>>>> *create) >>>>>>>> { >>>>>>>> unsigned requested_planes = 1; >>>>>>>> diff --git a/drivers/media/v4l2-core/v4l2-dev.c >>>>>>>> b/drivers/media/v4l2-core/v4l2-dev.c >>>>>>>> index f81279492682..80ace2e1e932 100644 >>>>>>>> --- a/drivers/media/v4l2-core/v4l2-dev.c >>>>>>>> +++ b/drivers/media/v4l2-core/v4l2-dev.c >>>>>>>> @@ -720,6 +720,7 @@ static void determine_valid_ioctls(struct >>>>>>>> video_device *vdev) >>>>>>>> SET_VALID_IOCTL(ops, VIDIOC_PREPARE_BUF, >>>>>>>> vidioc_prepare_buf); >>>>>>>> SET_VALID_IOCTL(ops, VIDIOC_STREAMON, >>>>>>>> vidioc_streamon); >>>>>>>> SET_VALID_IOCTL(ops, VIDIOC_STREAMOFF, >>>>>>>> vidioc_streamoff); >>>>>>>> + SET_VALID_IOCTL(ops, VIDIOC_DELETE_BUF, >>>>>>>> vidioc_delete_buf); >>>>>>>> } >>>>>>>> >>>>>>>> if (is_vid || is_vbi || is_meta) { >>>>>>>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>>>> b/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>>>> index a858acea6547..1c737279d3ef 100644 >>>>>>>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>>>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c >>>>>>>> @@ -2156,6 +2156,15 @@ static int v4l_prepare_buf(const struct >>>>>>>> v4l2_ioctl_ops *ops, >>>>>>>> return ret ? ret : ops->vidioc_prepare_buf(file, fh, b); >>>>>>>> } >>>>>>>> >>>>>>>> +static int v4l_delete_buf(const struct v4l2_ioctl_ops *ops, >>>>>>>> + struct file *file, void *fh, void *arg) >>>>>>>> +{ >>>>>>>> + struct v4l2_buffer *b = arg; >>>>>>>> + int ret = check_fmt(file, b->type); >>>>>>>> + >>>>>>>> + return ret ? ret : ops->vidioc_delete_buf(file, fh, b); >>>>>>>> +} >>>>>>>> + >>>>>>>> static int v4l_g_parm(const struct v4l2_ioctl_ops *ops, >>>>>>>> struct file *file, void *fh, void >>>>>>>> *arg) >>>>>>>> { >>>>>>>> @@ -2905,6 +2914,7 @@ static const struct v4l2_ioctl_info >>>>>>>> v4l2_ioctls[] = { >>>>>>>> IOCTL_INFO(VIDIOC_ENUM_FREQ_BANDS, v4l_enum_freq_bands, >>>>>>>> v4l_print_freq_band, 0), >>>>>>>> IOCTL_INFO(VIDIOC_DBG_G_CHIP_INFO, v4l_dbg_g_chip_info, >>>>>>>> v4l_print_dbg_chip_info, INFO_FL_CLEAR(v4l2_dbg_chip_info, match)), >>>>>>>> IOCTL_INFO(VIDIOC_QUERY_EXT_CTRL, v4l_query_ext_ctrl, >>>>>>>> v4l_print_query_ext_ctrl, INFO_FL_CTRL | >>>>>>>> INFO_FL_CLEAR(v4l2_query_ext_ctrl, id)), >>>>>>>> + IOCTL_INFO(VIDIOC_DELETE_BUF, v4l_delete_buf, >>>>>>>> v4l_print_buffer, INFO_FL_QUEUE), >>>>>>>> }; >>>>>>>> #define V4L2_IOCTLS ARRAY_SIZE(v4l2_ioctls) >>>>>>>> >>>>>>>> diff --git a/include/media/v4l2-ioctl.h b/include/media/v4l2-ioctl.h >>>>>>>> index edb733f21604..2f232ed884c7 100644 >>>>>>>> --- a/include/media/v4l2-ioctl.h >>>>>>>> +++ b/include/media/v4l2-ioctl.h >>>>>>>> @@ -163,6 +163,8 @@ struct v4l2_fh; >>>>>>>> * :ref:`VIDIOC_CREATE_BUFS <vidioc_create_bufs>` ioctl >>>>>>>> * @vidioc_prepare_buf: pointer to the function that implements >>>>>>>> * :ref:`VIDIOC_PREPARE_BUF <vidioc_prepare_buf>` ioctl >>>>>>>> + * @vidioc_delete_buf: pointer to the function that implements >>>>>>>> + * :ref:`VIDIOC_DELETE_BUF <vidioc_delete_buf>` ioctl >>>>>>>> * @vidioc_overlay: pointer to the function that implements >>>>>>>> * :ref:`VIDIOC_OVERLAY <vidioc_overlay>` ioctl >>>>>>>> * @vidioc_g_fbuf: pointer to the function that implements >>>>>>>> @@ -422,6 +424,8 @@ struct v4l2_ioctl_ops { >>>>>>>> struct v4l2_create_buffers *b); >>>>>>>> int (*vidioc_prepare_buf)(struct file *file, void *fh, >>>>>>>> struct v4l2_buffer *b); >>>>>>>> + int (*vidioc_delete_buf)(struct file *file, void *fh, >>>>>>>> + struct v4l2_buffer *b); >>>>>>>> >>>>>>>> int (*vidioc_overlay)(struct file *file, void *fh, unsigned >>>>>>>> int i); >>>>>>>> int (*vidioc_g_fbuf)(struct file *file, void *fh, >>>>>>>> diff --git a/include/media/videobuf2-core.h >>>>>>>> b/include/media/videobuf2-core.h >>>>>>>> index 080b783d608d..0f9e68f76b77 100644 >>>>>>>> --- a/include/media/videobuf2-core.h >>>>>>>> +++ b/include/media/videobuf2-core.h >>>>>>>> @@ -840,6 +840,15 @@ int vb2_core_create_bufs(struct vb2_queue *q, >>>>>>>> enum vb2_memory memory, >>>>>>>> */ >>>>>>>> int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, >>>>>>>> void *pb); >>>>>>>> >>>>>>>> +/** >>>>>>>> + * vb2_core_delete_buf() - >>>>>>>> + * @q: pointer to &struct vb2_queue with videobuf2 queue. >>>>>>>> + * @index: id number of the buffer. >>>>>>>> + * >>>>>>>> + * Return: returns zero on success; an error code otherwise. >>>>>>>> + */ >>>>>>>> +int vb2_core_delete_buf(struct vb2_queue *q, unsigned int index); >>>>>>>> + >>>>>>>> /** >>>>>>>> * vb2_core_qbuf() - Queue a buffer from userspace >>>>>>>> * >>>>>>>> diff --git a/include/media/videobuf2-v4l2.h >>>>>>>> b/include/media/videobuf2-v4l2.h >>>>>>>> index 88a7a565170e..3beeb4c735f0 100644 >>>>>>>> --- a/include/media/videobuf2-v4l2.h >>>>>>>> +++ b/include/media/videobuf2-v4l2.h >>>>>>>> @@ -114,6 +114,17 @@ int vb2_create_bufs(struct vb2_queue *q, struct >>>>>>>> v4l2_create_buffers *create); >>>>>>>> */ >>>>>>>> int vb2_prepare_buf(struct vb2_queue *q, struct media_device *mdev, >>>>>>>> struct v4l2_buffer *b); >>>>>>>> +/** >>>>>>>> + * vb2_delete_buf() - Delete the buffer from the queue >>>>>>>> + * >>>>>>>> + * @q: pointer to &struct vb2_queue with videobuf2 queue. >>>>>>>> + * @b: buffer structure passed from userspace to >>>>>>>> + * &v4l2_ioctl_ops->vidioc_delete_buf handler in driver >>>>>>>> + * >>>>>>>> + * The return values from this function are intended to be directly >>>>>>>> returned >>>>>>>> + * from &v4l2_ioctl_ops->vidioc_delete_buf handler in driver. >>>>>>>> + */ >>>>>>>> +int vb2_delete_buf(struct vb2_queue *q, struct v4l2_buffer *b); >>>>>>>> >>>>>>>> /** >>>>>>>> * vb2_qbuf() - Queue a buffer from userspace >>>>>>>> diff --git a/include/uapi/linux/videodev2.h >>>>>>>> b/include/uapi/linux/videodev2.h >>>>>>>> index aee75eb9e686..31bba1915642 100644 >>>>>>>> --- a/include/uapi/linux/videodev2.h >>>>>>>> +++ b/include/uapi/linux/videodev2.h >>>>>>>> @@ -2702,6 +2702,8 @@ struct v4l2_create_buffers { >>>>>>>> #define VIDIOC_DBG_G_CHIP_INFO _IOWR('V', 102, struct >>>>>>>> v4l2_dbg_chip_info) >>>>>>>> >>>>>>>> #define VIDIOC_QUERY_EXT_CTRL _IOWR('V', 103, struct >>>>>>>> v4l2_query_ext_ctrl) >>>>>>>> +#define VIDIOC_DELETE_BUF _IOWR('V', 104, struct v4l2_buffer) >>>>>>>> + >>>>>>>> >>>>>>>> /* Reminder: when adding new ioctls please add support for them to >>>>>>>> drivers/media/v4l2-core/v4l2-compat-ioctl32.c as well! */ >>>>>>>> -- >>>>>>>> 2.39.2 >>>>>>>> >>>>> -- >>>>> Hsia-Jun(Randy) Li >>>>> >>>> -- >>>> Hsia-Jun(Randy) Li >>>> >> -- >> Hsia-Jun(Randy) Li >> -- Hsia-Jun(Randy) Li ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v3 10/11] media: v4l2: Add DELETE_BUF ioctl [not found] ` <20230622131349.144160-11-benjamin.gaignard@collabora.com> [not found] ` <80a03c29-6f3d-43f1-755c-10f3ae2c2756@synaptics.com> @ 2023-07-02 10:20 ` Markus Elfring 1 sibling, 0 replies; 27+ messages in thread From: Markus Elfring @ 2023-07-02 10:20 UTC (permalink / raw) To: Benjamin Gaignard, linux-staging, linux-media, linux-arm-kernel, linux-arm-msm, linux-rockchip, linux-mediatek, kernel-janitors, Ezequiel Garcia, Greg Kroah-Hartman, Hans Verkuil, Marek Szyprowski, Mauro Carvalho Chehab, Ming Qian, Nicolas Dufresne, Philipp Zabel, Tomasz Figa Cc: LKML, kernel > VIDIOC_DELETE_BUF ioctl allows to delete a buffer from a queue. Please choose another imperative change suggestion. See also: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/process/submitting-patches.rst?h=v6.4#n94 Regards, Markus ^ permalink raw reply [flat|nested] 27+ messages in thread
end of thread, other threads:[~2023-09-07 6:54 UTC | newest]
Thread overview: 27+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <20230622131349.144160-1-benjamin.gaignard@collabora.com>
[not found] ` <20230622131349.144160-5-benjamin.gaignard@collabora.com>
2023-06-30 9:51 ` [PATCH v3 04/11] media: videobuf2: Stop define VB2_MAX_FRAME as global Hsia-Jun Li
2023-07-03 8:09 ` Benjamin Gaignard
2023-07-03 8:35 ` Hsia-Jun Li
2023-07-03 10:53 ` Benjamin Gaignard
2023-07-03 11:05 ` Hsia-Jun Li
2023-07-12 10:48 ` Tomasz Figa
2023-07-17 7:44 ` Hsia-Jun Li
2023-07-28 6:46 ` Tomasz Figa
2023-07-28 6:55 ` Hsia-Jun Li
2023-07-28 7:03 ` Tomasz Figa
2023-07-28 7:24 ` Hsia-Jun Li
2023-09-07 4:15 ` Tomasz Figa
2023-09-07 6:54 ` Hsia-Jun Li
[not found] ` <20230622131349.144160-11-benjamin.gaignard@collabora.com>
[not found] ` <80a03c29-6f3d-43f1-755c-10f3ae2c2756@synaptics.com>
[not found] ` <42a45bad-09c7-ffb6-49cb-29ec0826599d@collabora.com>
[not found] ` <3b22fda9-4a0e-2c03-2f12-19e5e3239235@synaptics.com>
2023-06-30 9:43 ` [PATCH v3 10/11] media: v4l2: Add DELETE_BUF ioctl Hsia-Jun Li
2023-07-03 8:12 ` Benjamin Gaignard
2023-07-03 8:19 ` Hsia-Jun Li
2023-07-03 8:52 ` Benjamin Gaignard
2023-07-03 9:20 ` Hsia-Jun Li
2023-07-03 10:35 ` Benjamin Gaignard
2023-07-03 11:06 ` Hsia-Jun Li
[not found] ` <8ca2f66e-8ff9-e885-274f-51417b581b78@synaptics.com>
2023-07-03 11:17 ` Benjamin Gaignard
2023-07-03 15:42 ` Randy Li
2023-07-13 9:09 ` Tomasz Figa
2023-07-17 2:16 ` Hsia-Jun Li
2023-07-28 6:57 ` Tomasz Figa
2023-07-28 7:26 ` Hsia-Jun Li
2023-07-02 10:20 ` Markus Elfring
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox