From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EA77EC3064D for ; Thu, 27 Jun 2024 11:11:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: Content-Type:In-Reply-To:From:References:Cc:To:Subject:MIME-Version:Date: Message-ID:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=RjG4NvLPZHP15XSF73Un+NgkQkULbEPKorgzxgkPLfo=; b=Vv7YQ6nZIjXcSd/tQ+2/MoYuVg hTOQGWqVE6UypSfe6YRP/dJ1POUSSEgWYivUOURoZC9IQ6oRCQeDbOtMlALa1xfRhK36UHxgL1jgf NIY8HMhDCc5gJrM906n9QCtralbeY1UcrzGb/zsaiWThVwVK61n6eoICFY4l7YC2CnHdH6XyvrudH 1CgxG4/ebbHkxIpNTKRdSOrMAMjnfF3dkpqxZhGiMU7e5nWgqK0Rkg5sjFPOM8o6kkXfR4Qw6zhF5 XXuDJb75ytpOl0MMNaZu4/X8K1fAg6ZtdTUE+1XgMBI9ow1NdD+N8+7rsF6BFAJQFsE/cOPunF9C1 CB++J7EA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sMn2K-0000000A6lN-1I1j; Thu, 27 Jun 2024 11:11:32 +0000 Received: from mail-ed1-x535.google.com ([2a00:1450:4864:20::535]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sMn1Z-0000000A6W4-0y2e; Thu, 27 Jun 2024 11:10:54 +0000 Received: by mail-ed1-x535.google.com with SMTP id 4fb4d7f45d1cf-57d26a4ee65so1354324a12.2; Thu, 27 Jun 2024 04:10:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1719486641; x=1720091441; darn=lists.infradead.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=RjG4NvLPZHP15XSF73Un+NgkQkULbEPKorgzxgkPLfo=; b=TJWc1E7aM4tjIa5XaKEghZuBEmpWevBc0/XssDla4mBA6BAzc743Lltacimf6JIjoA Ei/11RofSEhRM77CJbEXAqVpUcsUxeOFT0kJcAeLgMSnjjNAyd6MOpVmJ7mn+kfOTtDU V2pupoZJzUVzry7YSda485geMpOxxsYX9m8mMe1AAXcA7QQNQeenikFA7AQyFZaOtzeM 7A68oMvZYwoqn/iPSDNwhjazjJM9P+xTkWhuaz5Z1/utWicrBYnSCuFk6F92HPMyO5nI jPt0pJWmGEj3x1Rj+1k6UMyEZ6DiudU0qCiFb9F+yumpPesYS6ZSG6oHx3v2xuhg/4/d KS0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719486641; x=1720091441; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=RjG4NvLPZHP15XSF73Un+NgkQkULbEPKorgzxgkPLfo=; b=UpfvrNC2qn71CGGA8vu8Y2VwoS2hw4fs4hXgKSfut2xiGiKsVZAR2AVJOTkVZRIEne CCjb8xqzubTJS2xWDg6Qb+ax4jN9u8KCfaqFeCYQQaLt+ZMt8XpqaJyXVInNLwG7sxri HQZHUVgH4MrqMaMI0c5PJuLfEpIFBMdbVZ7B69YMvl7DNJVKlg6Kpls089MJDqD5f8Om g0CohWVk+cI/KTl/ePlkr+kUqHTG7tOZM/qKq6DHvl+SshOH4ROz6KkKY2nozbnQct5B mW0RJ8hawPdQ75b705G0ZIXYSPq/7I+VFLPIsTQuWLR93rMR3ck9I9nNvgVO16Me7kqI EjEQ== X-Forwarded-Encrypted: i=1; AJvYcCWMnx5nrRuffRtejjLzAOHjWSrLFrtzpljOTXKlvfbw0QaJvjuSDgFBZSeHfTptOEsXBVrsiPaVp8/6Wnc+zRmuDkOZNIWZGQn3tIXdcuyQfiZ6fcTJYlqwNFco1BISyARZ+OWvN6QZ2cOyFYyrAP5ounmJE5wsTr0= X-Gm-Message-State: AOJu0Yzxy/7GmIlnLUiZ2c9/RNErh69SuaxLCwwOU6fbHjaIq4LqBnQz mad0OmsFpj7yi0MP6Mz5kL+s3E+W1xHgQBvpZK814ILajrak5rU= X-Google-Smtp-Source: AGHT+IH4fbvY1cFOwHHzrwt3dVlavsf75UZvpMeelUbWWYiwXGXJ5CcSyvDpzggFY+YcW3P4/NbQ2g== X-Received: by 2002:a05:6402:520c:b0:582:1f66:1472 with SMTP id 4fb4d7f45d1cf-5821f661646mr6043539a12.31.1719486640245; Thu, 27 Jun 2024 04:10:40 -0700 (PDT) Received: from ?IPV6:2a02:810b:f40:4600:2023:48bf:6b4:88a4? ([2a02:810b:f40:4600:2023:48bf:6b4:88a4]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-584d16bbf1csm733229a12.44.2024.06.27.04.10.39 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 27 Jun 2024 04:10:39 -0700 (PDT) Message-ID: Date: Thu, 27 Jun 2024 13:10:38 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v3 2/4] media: rockchip: Introduce the rkvdec2 driver To: Jonas Karlman , Detlev Casanova Cc: Ezequiel Garcia , Mauro Carvalho Chehab , Rob Herring , Krzysztof Kozlowski , Conor Dooley , Heiko Stuebner , Greg Kroah-Hartman , Sebastian Reichel , Dragan Simic , Diederik de Haas , Andy Yan , Boris Brezillon , Hans Verkuil , Daniel Almeida , Paul Kocialkowski , Nicolas Dufresne , Benjamin Gaignard , linux-media@vger.kernel.org, linux-rockchip@lists.infradead.org, devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-staging@lists.linux.dev, linux-kernel@vger.kernel.org References: <20240620142532.406564-1-detlev.casanova@collabora.com> <20240620142532.406564-3-detlev.casanova@collabora.com> <3815203.kQq0lBPeGt@arisu> <5a15b138-4e03-4487-8a53-b7ff3527701f@gmail.com> <66e56996-4b56-4262-973f-672121071391@kwiboo.se> Content-Language: en-US From: Alex Bee In-Reply-To: <66e56996-4b56-4262-973f-672121071391@kwiboo.se> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240627_041045_332374_E67E551C X-CRM114-Status: GOOD ( 33.23 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Am 27.06.24 um 11:41 schrieb Jonas Karlman: > Hi Alex, > > On 2024-06-26 11:12, Alex Bee wrote: >> Hi Detlev, >> >> Am 25.06.24 um 18:56 schrieb Detlev Casanova: >>> Hi Alex, >>> >>> On Sunday, June 23, 2024 5:33:28 A.M. EDT you wrote: >>>> Hi Detlev, >>>> >>>> Am 20.06.24 um 16:19 schrieb Detlev Casanova: >>>>> This driver supports the second generation of the Rockchip Video >>>>> decoder, also known as vdpu34x. >>>>> It is currently only used on the RK3588(s) SoC. >>>>> >>>>> There are 2 decoders on the RK3588 SoC that can work in pair to decode >>>>> 8K video at 30 FPS but currently, only using one core at a time is >>>>> supported. >>>>> >>>>> Scheduling requests between the two cores will be implemented later. >>>>> >>>>> The core supports H264, HEVC, VP9 and AVS2 decoding but this driver >>>>> currently only supports H264. >>>>> >>>>> The driver is based on rkvdec and they may share some code in the >>>>> future. >>>>> The decision to make a different driver is mainly because rkvdec2 has >>>>> more features and can work with multiple cores. >>>>> >>>>> The registers are mapped in a struct in RAM using bitfields. It is IO >>>>> copied to the HW when all values are configured. >>>>> The decision to use such a struct instead of writing buffers one by one >>>>> >>>>> is based on the following reasons: >>>>> - Rockchip cores are known to misbehave when registers are not written >>>>> >>>>> in address order, >>>>> >>>>> - Those cores also need the software to write all registers, even if >>>>> >>>>> they are written their default values or are not related to the task >>>>> (this core will not start decoding some H264 frames if some VP9 >>>>> registers are not written to 0) >>>>> >>>>> - In the future, to support multiple cores, the scheduler could be >>>>> >>>>> optimized by storing the precomputed registers values and copy them >>>>> to the HW as soos as a core becomes available. >>>>> >>>>> This makes the code more readable and may bring performance improvements >>>>> in future features. >>>>> >>>>> Signed-off-by: Detlev Casanova >>>>> --- >>>>> >>>>> drivers/staging/media/Kconfig | 1 + >>>>> drivers/staging/media/Makefile | 1 + >>>>> drivers/staging/media/rkvdec2/Kconfig | 15 + >>>>> drivers/staging/media/rkvdec2/Makefile | 3 + >>>>> drivers/staging/media/rkvdec2/TODO | 9 + >>>>> drivers/staging/media/rkvdec2/rkvdec2-h264.c | 739 +++++++++++ >>>>> drivers/staging/media/rkvdec2/rkvdec2-regs.h | 345 +++++ >>>>> drivers/staging/media/rkvdec2/rkvdec2.c | 1253 ++++++++++++++++++ >>>>> drivers/staging/media/rkvdec2/rkvdec2.h | 130 ++ >>>>> 9 files changed, 2496 insertions(+) >>>>> create mode 100644 drivers/staging/media/rkvdec2/Kconfig >>>>> create mode 100644 drivers/staging/media/rkvdec2/Makefile >>>>> create mode 100644 drivers/staging/media/rkvdec2/TODO >>>>> create mode 100644 drivers/staging/media/rkvdec2/rkvdec2-h264.c >>>>> create mode 100644 drivers/staging/media/rkvdec2/rkvdec2-regs.h >>>>> create mode 100644 drivers/staging/media/rkvdec2/rkvdec2.c >>>>> create mode 100644 drivers/staging/media/rkvdec2/rkvdec2.h >>>> ... >>>> >>>>> +static inline void rkvdec2_memcpy_toio(void __iomem *dst, void *src, >>>>> size_t len) +{ >>>>> +#ifdef CONFIG_ARM64 >>>>> + __iowrite32_copy(dst, src, len); >>>>> +#elif defined(CONFIG_ARM) >>>> I guess that can get an "#else" since memcpy_toio exists for all archs. >>>> >>>>> + memcpy_toio(dst, src, len); >>>>> +#endif >>>>> +} >>>>> + >>>> ... >>>> >>>>> + /* Set timeout threshold */ >>>>> + if (pixels < RKVDEC2_1080P_PIXELS) >>>>> + regs->common.timeout_threshold = RKVDEC2_TIMEOUT_1080p; >>>>> + else if (pixels < RKVDEC2_4K_PIXELS) >>>>> + regs->common.timeout_threshold = RKVDEC2_TIMEOUT_4K; >>>>> + else if (pixels < RKVDEC2_8K_PIXELS) >>>>> + regs->common.timeout_threshold = RKVDEC2_TIMEOUT_8K; >>>>> + >>>> Did you test if it works with anything > 8K? If so, you propably want to >>>> make the check above >>>> >>>> + else >>>> + regs->common.timeout_threshold = RKVDEC2_TIMEOUT_8K; >>>> >>>> Otherwise the timeout may not be set/contain invalid values from any former >>>> stream. >>> That's right, but it would be set to 0 because of the memset. >>> RKVDEC2_TIMEOUT_8K might not be enough for bigger frame sizes, so I'll set it >>> to the maximum value (0xffffffff) when frames are bigger than 8K and also adapt >>> the watchdog time: RKVDEC2_TIMEOUT_8K is around 100 ms, but 0xffffffff is arnoud >>> 5.3 seconds (reg032/axi_clock_freq) >>> >>> I'll do more tests with this as well. >>> >>>> ... >>>> >>>>> + >>>>> +static const struct rkvdec2_coded_fmt_desc rkvdec2_coded_fmts[] = { >>>>> + { >>>>> + .fourcc = V4L2_PIX_FMT_H264_SLICE, >>>>> + .frmsize = { >>>>> + .min_width = 16, >>>>> + .max_width = 65520, >>>>> + .step_width = 16, >>>>> + .min_height = 16, >>>>> + .max_height = 65520, >>>>> + .step_height = 16, >>>>> + }, >>>>> + .ctrls = &rkvdec2_h264_ctrls, >>>>> + .ops = &rkvdec2_h264_fmt_ops, >>>>> + .num_decoded_fmts = >>> ARRAY_SIZE(rkvdec2_h264_decoded_fmts), >>>>> + .decoded_fmts = rkvdec2_h264_decoded_fmts, >>>>> + .subsystem_flags = >>> VB2_V4L2_FL_SUPPORTS_M2M_HOLD_CAPTURE_BUF, >>>>> + }, >>>>> +}; >>>>> + >>>> Note, that this is also given to userspace (VIDIOC_ENUM_FRAMESIZES) and >>>> this is already incorrect in the old rkvdec driver (and hantro): From >>>> userspace perspective we do not have a restriction in >>>> step_width/step_width, as we are aligning any given width/height to HW >>>> requirements in the driver - what we should give to userspace is >>>> fsize->type = V4L2_FRMSIZE_TYPE_CONTINUOUS; fsize->stepwise.min_height = >>>> 1; fsize->stepwise.min_width = 1; fsize->stepwise.max_height = 65520; >>>> fsize->stepwise.max_width = 65520; >>> Is fsize->stepwise.min_height = 1; and fsize->stepwise.min_width = 1 correct ? >>> Or do you mean fsize->stepwise.step_height = 1; and fsize->stepwise.setp_width >>> = 1 ? >>> >>> It would give this instead: >>> >>> .frmsize = { >>> .min_width = 16, >>> .max_width = 65520, >>> .step_width = 1, >>> .min_height = 16, >>> .max_height = 65520, >>> .step_height = 1, >>> }, >>> >>> and .vidioc_enum_framesizes sets fsize->type = V4L2_FRMSIZE_TYPE_CONTINUOUS; >> You can't adapt this here, because this .frmsize is also given to the >> v4l2_apply_frmsize_constraints helper, which does the actual alignment to >> HW requirements and requires the HW step_with and step_height. >> IIRC, we also align framesizes which are below minimum HW requirement, at >> least in rkvdec1 driver and it looks a lot like this is done here the same: >> so this should be .min_height = 1 and .min_width = 1. (I remember because >> there are VP9 conformance tests with very small framesizes). And yes, it >> looks like you've had to set .step_width and .step_height to 1 for >> V4L2_FRMSIZE_TYPE_CONTINUOUS, not sure why that is required. >> >> So, imho, the final rkvdec2_enum_framesizes should look like >> >> +static int rkvdec2_enum_framesizes(struct file *file, void *priv, >> +                   struct v4l2_frmsizeenum *fsize) >> .... >> +    fmt = rkvdec2_find_coded_fmt_desc(fsize->pixel_format); >> +    if (!fmt) >> +        return -EINVAL; >> + >> +    fsize->type = V4L2_FRMSIZE_TYPE_CONTINUOUS; >> +    fsize->stepwise.min_height = 1; >> +    fsize->stepwise.max_height = fmt->frmsize.max_height; >> +    fsize->stepwise.min_width = 1; >> +    fsize->stepwise.max_width = fmt->frmsize.max_width; >> +    fsize->stepwise.min_width = 1; >> +    fsize->stepwise.step_height = 1; >> +    fsize->stepwise.step_width = 1; >> +    return 0; >> +} >> >> Note: Not even build tested :) >> Jonas: maybe you can add a fixup patch to your rkvdec patches as well. > Thanks, will take a closer look and include something for rkvdec high10 > v6 later today/tomorrow. Great, thanks. To back up the "IIRC" part of my previous reply about .min_width and .min_height: v4l2_apply_frmsize_constraints does clamp_roundup on width and height, which means it sets height = hw_min_height and/or width = hw_min_width if the given width/height is smaller. Thus userspace doesn't need to care about width/height as long as it's 1 <= hw_max_height/hw_max_width. Alex > Regards, > Jonas > >> Regards, >> >> Alex >> >>>> I guess this new driver should be an >>>> opportunity to fix that and distinguish between internal and external >>>> frame size requirements and the .vidioc_enum_framesizes callback should >>>> adapted accordingly. Regards, Alex >>> Detlev.