From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E7F35C30653 for ; Wed, 26 Jun 2024 09:14:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: Content-Type:In-Reply-To:From:References:Cc:To:Subject:MIME-Version:Date: Message-ID:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=FongEw4DVTz7Uk7t+o0KVv9EUD1iaAZOmKGMy2sDWqU=; b=4Q9imMGmkGq4tUhju3+WRiKjgi 6fB/j0aMD1bXbtFJ5o3RhtsdeNh/kv00iYRhfKOVoL9ywPaH9lKErFtF3vk8FS0W2MrTbDNSHVmhZ 3f17eY8NJBXpmphKhvt9vrp8seCtyTLHCV9ZUyOPND01el7XV4LhqeJaIu/JAnI11dt/imu5ADlCT BEnpzgsE3O1NVULqTaixbrHCloiAZKu5GXS2YNE7PLhJJmz8W7elepRYMs1hE4PN7KfeMY3SAh+b+ HtyAjkxTjQKmTIHwAaE/R6O+ME9vPXTAlDmyYFAX13iMtjDyCM0saFM3SWpKNgI5gU4HIa+ErOAV4 WCEGPVaw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sMOjl-0000000662Q-0caB; Wed, 26 Jun 2024 09:14:45 +0000 Received: from mail-ed1-x531.google.com ([2a00:1450:4864:20::531]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sMOhP-000000065Nz-0GHL; Wed, 26 Jun 2024 09:12:20 +0000 Received: by mail-ed1-x531.google.com with SMTP id 4fb4d7f45d1cf-57d0699fd02so3615949a12.0; Wed, 26 Jun 2024 02:12:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1719393135; x=1719997935; darn=lists.infradead.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=FongEw4DVTz7Uk7t+o0KVv9EUD1iaAZOmKGMy2sDWqU=; b=Llk9KIP0DoQp8U2DabWo7t65xj4EXL6YoFnfXqj1pvTaSS15pTczr/y2iwDXXcTUXk Ov5XdGwxvdX68Z8ZRpFw/9wR4Z4z50WZmHYfFfZ7dJpWdItEljyY8PGbWQyi1EQt3x6+ 3759CAAJyzlm/wcMkbVZ0ERu4ELtKWp0NodsB42R5xdEPdVAseIeyJI6MSsz8854Ecrq 3y8WozRJuC84suQKrdmhGhcUQi6bwKHhPzNcyP2fkWwx0WiuxHOr2aeKy3BNFEP7uOcw CIMwx2cRH2Jw1dCNhZHgQMMCbPzsp5s1YZaaPRkXPmnTrTWSh2pZNrg5cWE7RhxKQ+fq 5GQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719393135; x=1719997935; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=FongEw4DVTz7Uk7t+o0KVv9EUD1iaAZOmKGMy2sDWqU=; b=Ptr8gdxEoeI7pXTSKwz/queaWRUKngvseptS8tqIsoo0Elthml0ha2k1sUWnM0oVMI AECbPW8zGqYIsmMoPDj18NcAFUWoRTllcHdEFjibNrBhR1hBNU+u3NzZ6fuSRB41RTkF xfs2yUJ6C3BtaYZ7DhBbkEmI9JuYql6sNRbrqJ3m0J0wSEhju5tXQjHbz1MZIGeCu8pA N9ODOphgZrtof7zOSPwDOtg4aAtlbjaWc0h1SoIJjX7h3GSu34Y5eeulutMBhzMgMLOg a/iRFG0Uku3lFjuTni0sSMXjh7L9IK/r/Rt+mQ1a0vQSKkRtS/VaViIS3lFkWQ6eWtuF b0OA== X-Forwarded-Encrypted: i=1; AJvYcCVTfRWTsGfztCHJesZZm3qNNe/JZeECVDb3K+3a4Sd7y14jZGOCdXxK54uiFwtMmMI7Y9yvFkho8VGCvUEqOI0XGucd6yt85KYcHFbhFrI8Qag3k8dMDf+E6wrg8wMThp8zGhWCGEQezavoGBHIaEzKRsbsH6pT0wo= X-Gm-Message-State: AOJu0Ywty8j7DMqMwVReFKOf2iT6p6beWRjJHwSmzuhh664Hmt8npGYR mTYvvpTocmU6fRF64fHe63H6+avPsNGxjuthNn+cE9sPcajdFaM= X-Google-Smtp-Source: AGHT+IGb5T6cpB/7EWvmhQ43Jf3bxUsfnnaogIExV5juuGP3ucFrS+SngbbnT41X1ojvJgNGssHk3Q== X-Received: by 2002:a50:d5d3:0:b0:57c:7c44:74df with SMTP id 4fb4d7f45d1cf-57d4bdcabb7mr8935735a12.29.1719393133298; Wed, 26 Jun 2024 02:12:13 -0700 (PDT) Received: from ?IPV6:2a02:810b:f40:4600:79e0:cc0d:71b1:3c08? ([2a02:810b:f40:4600:79e0:cc0d:71b1:3c08]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-57d30413446sm7045617a12.31.2024.06.26.02.12.12 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 26 Jun 2024 02:12:12 -0700 (PDT) Message-ID: <5a15b138-4e03-4487-8a53-b7ff3527701f@gmail.com> Date: Wed, 26 Jun 2024 11:12:13 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v3 2/4] media: rockchip: Introduce the rkvdec2 driver To: Detlev Casanova , linux-kernel@vger.kernel.org Cc: Ezequiel Garcia , Mauro Carvalho Chehab , Rob Herring , Krzysztof Kozlowski , Conor Dooley , Heiko Stuebner , Greg Kroah-Hartman , Sebastian Reichel , Dragan Simic , Diederik de Haas , Andy Yan , Boris Brezillon , Hans Verkuil , Daniel Almeida , Paul Kocialkowski , Nicolas Dufresne , Benjamin Gaignard , Jonas Karlman , linux-media@vger.kernel.org, linux-rockchip@lists.infradead.org, devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-staging@lists.linux.dev References: <20240620142532.406564-1-detlev.casanova@collabora.com> <20240620142532.406564-3-detlev.casanova@collabora.com> <3815203.kQq0lBPeGt@arisu> Content-Language: en-US From: Alex Bee In-Reply-To: <3815203.kQq0lBPeGt@arisu> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240626_021219_256278_135C8DFB X-CRM114-Status: GOOD ( 38.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi Detlev, Am 25.06.24 um 18:56 schrieb Detlev Casanova: > Hi Alex, > > On Sunday, June 23, 2024 5:33:28 A.M. EDT you wrote: >> Hi Detlev, >> >> Am 20.06.24 um 16:19 schrieb Detlev Casanova: >>> This driver supports the second generation of the Rockchip Video >>> decoder, also known as vdpu34x. >>> It is currently only used on the RK3588(s) SoC. >>> >>> There are 2 decoders on the RK3588 SoC that can work in pair to decode >>> 8K video at 30 FPS but currently, only using one core at a time is >>> supported. >>> >>> Scheduling requests between the two cores will be implemented later. >>> >>> The core supports H264, HEVC, VP9 and AVS2 decoding but this driver >>> currently only supports H264. >>> >>> The driver is based on rkvdec and they may share some code in the >>> future. >>> The decision to make a different driver is mainly because rkvdec2 has >>> more features and can work with multiple cores. >>> >>> The registers are mapped in a struct in RAM using bitfields. It is IO >>> copied to the HW when all values are configured. >>> The decision to use such a struct instead of writing buffers one by one >>> >>> is based on the following reasons: >>> - Rockchip cores are known to misbehave when registers are not written >>> >>> in address order, >>> >>> - Those cores also need the software to write all registers, even if >>> >>> they are written their default values or are not related to the task >>> (this core will not start decoding some H264 frames if some VP9 >>> registers are not written to 0) >>> >>> - In the future, to support multiple cores, the scheduler could be >>> >>> optimized by storing the precomputed registers values and copy them >>> to the HW as soos as a core becomes available. >>> >>> This makes the code more readable and may bring performance improvements >>> in future features. >>> >>> Signed-off-by: Detlev Casanova >>> --- >>> >>> drivers/staging/media/Kconfig | 1 + >>> drivers/staging/media/Makefile | 1 + >>> drivers/staging/media/rkvdec2/Kconfig | 15 + >>> drivers/staging/media/rkvdec2/Makefile | 3 + >>> drivers/staging/media/rkvdec2/TODO | 9 + >>> drivers/staging/media/rkvdec2/rkvdec2-h264.c | 739 +++++++++++ >>> drivers/staging/media/rkvdec2/rkvdec2-regs.h | 345 +++++ >>> drivers/staging/media/rkvdec2/rkvdec2.c | 1253 ++++++++++++++++++ >>> drivers/staging/media/rkvdec2/rkvdec2.h | 130 ++ >>> 9 files changed, 2496 insertions(+) >>> create mode 100644 drivers/staging/media/rkvdec2/Kconfig >>> create mode 100644 drivers/staging/media/rkvdec2/Makefile >>> create mode 100644 drivers/staging/media/rkvdec2/TODO >>> create mode 100644 drivers/staging/media/rkvdec2/rkvdec2-h264.c >>> create mode 100644 drivers/staging/media/rkvdec2/rkvdec2-regs.h >>> create mode 100644 drivers/staging/media/rkvdec2/rkvdec2.c >>> create mode 100644 drivers/staging/media/rkvdec2/rkvdec2.h >> ... >> >>> +static inline void rkvdec2_memcpy_toio(void __iomem *dst, void *src, >>> size_t len) +{ >>> +#ifdef CONFIG_ARM64 >>> + __iowrite32_copy(dst, src, len); >>> +#elif defined(CONFIG_ARM) >> I guess that can get an "#else" since memcpy_toio exists for all archs. >> >>> + memcpy_toio(dst, src, len); >>> +#endif >>> +} >>> + >> ... >> >>> + /* Set timeout threshold */ >>> + if (pixels < RKVDEC2_1080P_PIXELS) >>> + regs->common.timeout_threshold = RKVDEC2_TIMEOUT_1080p; >>> + else if (pixels < RKVDEC2_4K_PIXELS) >>> + regs->common.timeout_threshold = RKVDEC2_TIMEOUT_4K; >>> + else if (pixels < RKVDEC2_8K_PIXELS) >>> + regs->common.timeout_threshold = RKVDEC2_TIMEOUT_8K; >>> + >> Did you test if it works with anything > 8K? If so, you propably want to >> make the check above >> >> + else >> + regs->common.timeout_threshold = RKVDEC2_TIMEOUT_8K; >> >> Otherwise the timeout may not be set/contain invalid values from any former >> stream. > That's right, but it would be set to 0 because of the memset. > RKVDEC2_TIMEOUT_8K might not be enough for bigger frame sizes, so I'll set it > to the maximum value (0xffffffff) when frames are bigger than 8K and also adapt > the watchdog time: RKVDEC2_TIMEOUT_8K is around 100 ms, but 0xffffffff is arnoud > 5.3 seconds (reg032/axi_clock_freq) > > I'll do more tests with this as well. > >> ... >> >>> + >>> +static const struct rkvdec2_coded_fmt_desc rkvdec2_coded_fmts[] = { >>> + { >>> + .fourcc = V4L2_PIX_FMT_H264_SLICE, >>> + .frmsize = { >>> + .min_width = 16, >>> + .max_width = 65520, >>> + .step_width = 16, >>> + .min_height = 16, >>> + .max_height = 65520, >>> + .step_height = 16, >>> + }, >>> + .ctrls = &rkvdec2_h264_ctrls, >>> + .ops = &rkvdec2_h264_fmt_ops, >>> + .num_decoded_fmts = > ARRAY_SIZE(rkvdec2_h264_decoded_fmts), >>> + .decoded_fmts = rkvdec2_h264_decoded_fmts, >>> + .subsystem_flags = > VB2_V4L2_FL_SUPPORTS_M2M_HOLD_CAPTURE_BUF, >>> + }, >>> +}; >>> + >> Note, that this is also given to userspace (VIDIOC_ENUM_FRAMESIZES) and >> this is already incorrect in the old rkvdec driver (and hantro): From >> userspace perspective we do not have a restriction in >> step_width/step_width, as we are aligning any given width/height to HW >> requirements in the driver - what we should give to userspace is >> fsize->type = V4L2_FRMSIZE_TYPE_CONTINUOUS; fsize->stepwise.min_height = >> 1; fsize->stepwise.min_width = 1; fsize->stepwise.max_height = 65520; >> fsize->stepwise.max_width = 65520; > Is fsize->stepwise.min_height = 1; and fsize->stepwise.min_width = 1 correct ? > Or do you mean fsize->stepwise.step_height = 1; and fsize->stepwise.setp_width > = 1 ? > > It would give this instead: > > .frmsize = { > .min_width = 16, > .max_width = 65520, > .step_width = 1, > .min_height = 16, > .max_height = 65520, > .step_height = 1, > }, > > and .vidioc_enum_framesizes sets fsize->type = V4L2_FRMSIZE_TYPE_CONTINUOUS; You can't adapt this here, because this .frmsize is also given to the v4l2_apply_frmsize_constraints helper, which does the actual alignment to HW requirements and requires the HW step_with and step_height. IIRC, we also align framesizes which are below minimum HW requirement, at least in rkvdec1 driver and it looks a lot like this is done here the same: so this should be .min_height = 1 and .min_width = 1. (I remember because there are VP9 conformance tests with very small framesizes). And yes, it looks like you've had to set .step_width and .step_height to 1 for V4L2_FRMSIZE_TYPE_CONTINUOUS, not sure why that is required. So, imho, the final rkvdec2_enum_framesizes should look like +static int rkvdec2_enum_framesizes(struct file *file, void *priv, +                   struct v4l2_frmsizeenum *fsize) .... +    fmt = rkvdec2_find_coded_fmt_desc(fsize->pixel_format); +    if (!fmt) +        return -EINVAL; + +    fsize->type = V4L2_FRMSIZE_TYPE_CONTINUOUS; +    fsize->stepwise.min_height = 1; +    fsize->stepwise.max_height = fmt->frmsize.max_height; +    fsize->stepwise.min_width = 1; +    fsize->stepwise.max_width = fmt->frmsize.max_width; +    fsize->stepwise.min_width = 1; +    fsize->stepwise.step_height = 1; +    fsize->stepwise.step_width = 1; +    return 0; +} Note: Not even build tested :) Jonas: maybe you can add a fixup patch to your rkvdec patches as well. Regards, Alex >> I guess this new driver should be an >> opportunity to fix that and distinguish between internal and external >> frame size requirements and the .vidioc_enum_framesizes callback should >> adapted accordingly. Regards, Alex > Detlev.