[RFC] Cropping problem with V4L2, crop extension

[Tomasz Stanislawski <t.stanislaws@samsung.com>]

Hi everyone,
I am working on HDMI driver for latest Samsung SoC. I encountered 
problem with format and crop setup. I noticed that this issue was 
mentioned other threads before. First I would like to describe 
environment of a cropping problem.
There are two pieces of multimedia hardware on board I am working with. 
The avi decoder called MFC, and hdmi hardware called HDMI.  Please refer 
to posts for further details:

http://thread.gmane.org/gmane.linux.drivers.video-input-infrastructure/24022
http://thread.gmane.org/gmane.linux.drivers.video-input-infrastructure/28885

The MFC transforms movie stream to image in NV12T format. This format is 
very similar to NV12 but pixels are organized into macroblocks of size 
64 x 32. More information about NV12T format can be found in post:
http://thread.gmane.org/gmane.linux.drivers.video-input-infrastructure/28942
Due to hardware limitations sizes of output image have to multiples of 
128 x 32 pixels. The problem comes when movie size is not a multiple of 
128 x 32. The MFC is a M2M device, so it supports G_CROP ioctl on both 
CAPTURE and OUTPUT buffer. The active area in output image is obtained 
by calling G_CROP on CAPTURE buffer.

Let's consider following scenario:

MFC configuration is done in the first few steps.
1. We have H264 movie in resolution 320 x 240.
2. ioctl(fd = MFC, S_FMT, v4l2_format { .type = OUTPUT,
    .pixelformat=FMT_H264})
3. Allocates buffers using REQBUF
4. QBUF few frames to get stream resolution
5. Calls ioctl(fd = MFC, G_FMT, v4l2_format { .type = CAPTURE}) to get 
width and height of buffer that suit to the movie. Lets call it 
full_width, and full_height. These are 384 x 256 because it is the 
smallest multiple of 128 x  64 that contains full image.
6. Call ioctl(fd = MFC, G_CROP, v4l2_crop { .type = CAPTURE}) to get 
movie resolution. Call it width and height.

Now HDMI is configured. Assume 720p resolution on display. One wants to 
see movie on full screen, this is resolution 1280 x 720.

7. ioctl(fd = HDMI, S_FMT, v4l2_format { .type = OUTPUT,
   .pixelformat=FMT_NV12T, .height = full_height, .width = full_width })
8. Allocate buffers using REQBUF
9. Queue buffers and start streaming and so on.

In such a case, MFC creates buffer of size 384 x 256. Only subimage of 
size 320 x 240 contains movie frame. All padding pixels are filled with 
zeros. The output image 384 x 256 is passed to HDMI which is an V4L2 
output device. Zeroed pixels in YCrCr color space it an ugly green 
color. Such an ugly color is seen on output display.

And here comes the problem.

---------------------------------------------------------------

How to eliminate those ugly padding pixels from display output?

---------------------------------------------------------------

I think that possible solution to such a problem would be introducing 
new functionality called buffer crop. Please look at figures below. 
Figure 1 is similar to image 1.1 from V4L2 documentation:
http://www.linuxtv.org/downloads/v4l-dvb-apis/crop.html#crop-scale.
The value bounds.* refer to v4l2_cropcap.bounds. These values describe 
limits for cropping at output. The values c.* refer to v4l2_crop.c and 
describe active area on output device. Figure 2 depicts new feature 
called buffer crop. The image sizes pix.width and pix.height are 
configured using VIDIOC_S_FMT ioctl. The values bc.* define cropping 
rectangle in buffer. Only pixels inside IMAGE DATA rectangle are 
written/read during streaming. Only these pixels should be passed
to HDMI output. All other pixels are left untouched.

       bounds.width
   +--------------------+
   |                    |
  b| (c.left, c.top)    |
  o|       /            |
  u|      /             |
  n|     / c.width      |
  d|    +---------+     |
  s|  c |         |     |
  .|  . |         |     |
  h|  h |         |     |
  e|  e |  IMAGE  |     |
  i|  i |   DATA  |     |
  g|  g |         |     |
  h|  h |         |     |
  t|  t +---------+     |
   |                    |
   |                    |
   +--------------------+

Fig 1. Data on display.


       pix.width
  +-----------------------+
  |                       |
  |(bc.left, bc.top)      |
p|        \              |
i|         \  bc.width   |
x|          +---------+  |
.|         b|         |  |
h|         c|         |  |
e|         .|         |  |
i|         h|  IMAGE  |  |
g|         e|   DATA  |  |
h|         i|         |  |
t|         g|         |  |
  |         h|         |  |
  |         t+---------+  |
  |                       |
  |                       |
  +-----------------------+

Fig 2. Data in buffer.

Going back to above-mentioned scenario described.  In order to avoid 
showing ugly pixels on output display one would configure buffer crop 
with value { left = 0, top = 0, width = 320, height = 240}. The driver 
would automatically adjust all values to accepted limits, align and 
scaling factors. For hw I possess no such changes need to be applied.

The only problem is passing such a buffer crop to driver by V4L2 api.
I considered a few solutions.

1. Use existing API.
Use bytesperline field in struct v4l2_pix_format to configure real width 
of image data. Additionally one would have to compute proper pointer in
v4l2_buffer.m.userptr to simulate left/top offsets. Unfortunately, value
bytesperline has little meaning for macroblock formats, because there is 
no such a thing like 'Distance in bytes between the leftmost pixels in 
two adjacent lines'. Pixels are not ordered linearly. Better definition 
of bytesperline has to be stated. Computing proper pointer to pixel for 
image in macroblock format would be possible only for very small set of 
coordinates. It may involve complex and error-prone computations.

What about more complex formats like JPEG? It is not possible to compute 
a pointer without decompressing image, and parsing it, and applying 
modification to data before passing buffer by VIDIOC_QBUF. It is not 
possible to perform such cropping even though it is support in hardware.

2. Use existing API. Version II.
The v4l2_crop structure lacks any free place for extensions. It is quite 
often that a device supports only one buffer type. Then other buffer 
types could be passed in v4l2_crop.type. For example passing 
V4L2_BUF_TYPE_VIDEO_OVERLAY for devices that support only 
V4L2_BUF_TYPE_VIDEO_OUTPUT would indicate configuration of buffer crop. 
I heard some rumour that Nokia once used this solution. For M2M devices 
buffer cropping in OUTPUT buffer is configured by setting crop on 
CAPTURE buffer and vice-versa.

3. Introduce new ioctl.
New ioctl called VIDIOC_S_BUFCROP would be used to configure buffer 
cropping. It would use exactly the same parameters as VIDIOC_S_CROP. 
Additionally VIDIOC_S_BUFCROPCAP would be added. It would be used to 
inform application about possible cropping in buffer data. Ioctl 
VIDIOC_G_BUFCROP would return current configuration of cropping. If 
driver does not support buffer cropping then ioctl would return {left = 
0, top = 0, width = image_width, height = image_height}.

4. Introduce new ioctl. Version II
There was much discussion on scaling configuration by V4L2 api. Please 
look to post:
http://thread.gmane.org/gmane.linux.drivers.video-input-infrastructure/27581/focus=28243
It was stated there that atomic crop/format configuration is needed. I 
thought that it would be a good idea to kill two birds using one stone. 
I mean setting both normal crop and buffer crop at the same time. Usage 
of buffer cropping may help to avoid changing image format. So following 
structure is proposed. It is passed to driver with new VIDIOC_S_CROP2 ioctl.

struct v4l2_crop2 {
	u32 type; /* buffer type */
	struct v4l2_crop c; /* crop in hw configured by S_CROP ioctl */
	struct v4l2_crop b; /* crop in buffer */
	u32 hint; /* hints for driver for crop adjustments */
	u32 reserved[?]; /* place for possible extensions */
};

Hint would be bit flags that Exemplary hint might be 
V4L2_CROP_HINT_B_WIDTH_UP, which would prevent driver from lowering 
width of buffer cropping. Keeping height on c crop fixed could be 
achieved by (V4L2_CROP_HINT_C_HEIGHT_UP | V4L2_CROP_HINT_C_HEIGHT_DOWN). 
Other hint might be V4L2_CROP_HINT_FMT_FIXED that would imply keeping 
width and height of image fixed. The driver would use hints to adjust 
cropping according both to application and hardware restriction
simultaneously.

What it your opinion about proposed solutions?

Looking for a reply,

Best regards,
Tomasz Stanislawski