From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE304C48BD5 for ; Tue, 25 Jun 2019 10:27:45 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8BB4B214DA for ; Tue, 25 Jun 2019 10:27:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="a6JQ15Vb" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8BB4B214DA Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=mediatek.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Date:To:From:Subject:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=nx+AWNQklamWqdk2zgosKMOFzqbjle/CHF7Fsq0AQsk=; b=a6JQ15Vb8Aq4sF BjXCmEx3aL7+kpJsuH8Ms4lY0/h0JTuldNzgf2r9CuK9otTyZNWAhcesZpOCtAnANZbcxmVFQjryl fIGXALzSWErJEerjvuJOA0SHYq9ObPDyJOIPew/rwjKQwBbmL4fHqCqTsu/AxqGH7jrS2vgoNE6k6 Sz7cC6KRYF50JfXsKpAwXJWllSxS2Hj5s2g9+qg4wb5InZm6at2anwLzbCwzR9YfcDqtJVoM3eh2O KFXmuZoQi7wPeR5bN4cBmHjjKUjbZPB6NSazhqAxbWZxB0BTKvboyxqnqJcNEMa+3kTYKXtQsO3xh lYg+bb4r6aauMjgja0Wg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92 #3 (Red Hat Linux)) id 1hfiek-0004YR-Nc; Tue, 25 Jun 2019 10:26:30 +0000 Received: from mailgw01.mediatek.com ([216.200.240.184]) by bombadil.infradead.org with esmtps (Exim 4.92 #3 (Red Hat Linux)) id 1hfidS-0003TR-NU; Tue, 25 Jun 2019 10:25:15 +0000 X-UUID: 0ca5d1a723ab49939ab9dac19cbdcf7e-20190625 X-UUID: 0ca5d1a723ab49939ab9dac19cbdcf7e-20190625 Received: from mtkcas66.mediatek.inc [(172.29.193.44)] by mailgw01.mediatek.com (envelope-from ) (musrelay.mediatek.com ESMTP with TLS) with ESMTP id 1350816109; Tue, 25 Jun 2019 02:24:59 -0800 Received: from MTKMBS01N1.mediatek.inc (172.21.101.68) by MTKMBS62N2.mediatek.inc (172.29.193.42) with Microsoft SMTP Server (TLS) id 15.0.1395.4; Tue, 25 Jun 2019 03:24:58 -0700 Received: from mtkcas08.mediatek.inc (172.21.101.126) by mtkmbs01n1.mediatek.inc (172.21.101.68) with Microsoft SMTP Server (TLS) id 15.0.1395.4; Tue, 25 Jun 2019 18:24:56 +0800 Received: from [172.21.84.99] (172.21.84.99) by mtkcas08.mediatek.inc (172.21.101.73) with Microsoft SMTP Server id 15.0.1395.4 via Frontend Transport; Tue, 25 Jun 2019 18:24:56 +0800 Message-ID: <1561458296.15267.263.camel@mtksdccf07> Subject: Re: [RFC PATCH V1 6/6] platform: mtk-isp: Add Mediatek FD driver From: Jerry-ch Chen To: Tomasz Figa Date: Tue, 25 Jun 2019 18:24:56 +0800 In-Reply-To: References: <20190423104505.38778-1-Jerry-Ch.chen@mediatek.com> <20190423104505.38778-7-Jerry-Ch.chen@mediatek.com> <20190606104347.GA107267@chromium.org> <1561386122.15267.223.camel@mtksdccf07> <1561452912.15267.261.camel@mtksdccf07> X-Mailer: Evolution 3.2.3-0ubuntu6 MIME-Version: 1.0 X-MTK: N X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190625_032510_898736_663501D9 X-CRM114-Status: GOOD ( 41.65 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "devicetree@vger.kernel.org" , Sean Cheng =?UTF-8?Q?=28=E9=84=AD=E6=98=87=E5=BC=98=29?= , "laurent.pinchart+renesas@ideasonboard.com" , Rynn Wu =?UTF-8?Q?=28=E5=90=B3=E8=82=B2=E6=81=A9=29?= , srv_heupstream , "mchehab@kernel.org" , Po-Yang Huang =?UTF-8?Q?=28=E9=BB=83=E6=9F=8F=E9=99=BD=29?= , Holmes Chiou =?UTF-8?Q?=28=E9=82=B1=E6=8C=BA=29?= , "suleiman@chromium.org" , "shik@chromium.org" , Jungo Lin =?UTF-8?Q?=28=E6=9E=97=E6=98=8E=E4=BF=8A=29?= , Sj Huang =?UTF-8?Q?=28=E9=BB=83=E4=BF=A1=E7=92=8B=29?= , "yuzhao@chromium.org" , "linux-mediatek@lists.infradead.org" , "zwisler@chromium.org" , "matthias.bgg@gmail.com" , Christie Yu =?UTF-8?Q?=28=E6=B8=B8=E9=9B=85=E6=83=A0=29?= , Frederic Chen =?UTF-8?Q?=28=E9=99=B3=E4=BF=8A=E5=85=83=29?= , "hans.verkuil@cisco.com" , "linux-arm-kernel@lists.infradead.org" , "linux-media@vger.kernel.org" Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi Tomasz, On Tue, 2019-06-25 at 18:09 +0800, Tomasz Figa wrote: > On Tue, Jun 25, 2019 at 5:55 PM Jerry-ch Chen > wrote: > > > > Hi Tomasz, > > > > On Tue, 2019-06-25 at 11:39 +0800, Tomasz Figa wrote: > > > Hi Jerry, > > > > > > On Mon, Jun 24, 2019 at 11:22 PM Jerry-ch Chen > > > wrote: > > > > > > > > Hi Tomasz, > > > > > > > > On Thu, 2019-06-06 at 18:43 +0800, Tomasz Figa wrote: > > > > > Hi Jerry, > > > > > > > > > > On Tue, Apr 23, 2019 at 06:45:05PM +0800, Jerry-ch Chen wrote: > > > > > > From: Jerry-ch Chen > > > > > > > > > > > > This patch adds the driver of Face Detection (FD) unit in > > > > > > Mediatek camera system, providing face detection function. > > > > > > > > > > > > The mtk-isp directory will contain drivers for multiple IP > > > > > > blocks found in Mediatek ISP system. It will include ISP Pass 1 > > > > > > driver (CAM), sensor interface driver, DIP driver and face > > > > > > detection driver. > > > > > > > > > > > > > > > > Thanks for the patch. > > > > > > > > > > First of all a general comment about the design: > > > > > > > > > > My understanding is that this is a relatively straightforward > > > > > memory-to-memory device that reads a video frame and detects faces on it. > > > > > Such devices should be implemented as normal V4L2 memory-to-memory devices, > > > > > with contexts (instances; pipes) represented by v4l2_fh. > > > > > > > > > > Also, please replace the META_OUTPUT queue with proper V4L2 controls, as I > > > > > don't think there is anything that we couldn't model using controls here. > > > > > > > > > > The end result should be a V4L2 m2m driver (using the m2m helpers), where > > > > > you get a new context (instance; pipe) whenever you open the video node, > > > > > similar to codecs, video processors (like MTK MDP) and so on. > > > > > > > > > > Also please see my comments inline. > > > > > > > > > I appreciate your comments, > > > > sorry for sending the previous two unfinished mail... > > > > > > > > FD driver will be implemented as a normal V4L2 m2m driver which has an > > > > IMAGE_OUTPUT queue and a META_CAPTURE queue(face result). > > > > > > > > We will use the following properties. > > > > /* Is a video mem-to-mem device that supports multiplanar formats */ > > > > #define V4L2_CAP_VIDEO_M2M_MPLANE 0x00004000 > > > > > > > > The original META_OUTPUT queue contains the following structure will be > > > > replaced by V4L2 controls, > > > > > > > > /* FD_SCALE_NUM is 15. */ > > > > struct fd_user_param { > > > > uint8_t rip_feature; > > > > uint8_t gfd_skip; > > > > uint8_t dynamic_change_model; > > > > uint8_t scale_num_from_user; > > > > uint16_t source_img_width[FD_SCALE_NUM]; > > > > uint16_t source_img_height[FD_SCALE_NUM]; > > > > } __packed; //share with co-processor > > > > > > > > However, we found that testM2MFormats() in the V4L2 compliance test will > > > > assume the capture queue has the same format as output queue has, > > > > therefore, FD driver's capture queue wouldn't be able to use META format > > > > or the v4l2 test will be failed. > > > > > > > > reference: v4l2-compliance/v4l2-test-formats.cpp > > > > // m2m devices are special in that the format is often per-filehandle. > > > > // But colorspace information should be passed from output to capture, > > > > // so test that. > > > > if (node->is_m2m) > > > > return testM2MFormats(node); > > > > > > > > May we ask for your suggestions about this part? > > > > > > > > > > Ah, I didn't mean mem-to-mem device specifically as per > > > V4L2_CAP_VIDEO_M2M_MPLANE, because that one implies the regular > > > VIDEO_OUTPUT -> VIDEO_CAPTURE processing indeed. We should expose just > > > VIDEO_OUTPUT_MPLANE and META_CAPTURE in the capabilities, but all the > > > rest would still behave like a mem-to-mem device, i.e. v4l2_fh for > > > contexts/instances, v4l2_m2m helpers and so on. > > > > > I Appreciate for your reply, > > > > Sorry I didn't mention the question clearly, we have included these two > > capabilities, but we get the following v4l2 test failure: > > fail: v4l2-test-formats.cpp(784): fmt_cap.g_colorspace() != > > fmt_out.g_colorspace() > > My point is that we shouldn't set V4L2_CAP_VIDEO_M2M(_MPLANE) in the > capabilities, because FD is mem-to-mem in terms of the mode of > operation, not in terms of the definition of > V4L2_CAP_VIDEO_M2M(_MPLANE). That would make testM2MFormats() not > called at all. > > Best regards, > Tomasz > Ok, we got your point, Thanks and Best Regards, Jerry > > > > Which is caused by the following code testing the m2m buffers' > > capabilities, FD driver have fmt_cap with V4L2_BUF_TYPE_META_CAPTURE and > > fmt_out with V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE, therefore, our fmt_cap > > won't have colorspace information. > > > > Reference: > > https://github.com/gjasny/v4l-utils/blob/master/utils/v4l2-compliance/v4l2-test-formats.cpp#L774 > > fail_on_test(fmt_cap.g_colorspace() != fmt_out.g_colorspace()); > > fail_on_test(fmt_cap.g_ycbcr_enc() != fmt_out.g_ycbcr_enc()); > > fail_on_test(fmt_cap.g_quantization() != fmt_out.g_quantization()); > > fail_on_test(fmt_cap.g_xfer_func() != fmt_out.g_xfer_func()); > > > > Not sure if the maintainer of v4l2 test would consider modifying here to > > allow the use case of FD driver? > > > > > [snip] > > > > > > > > > + > > > > > > + return 0; > > > > > > +} > > > > > > + > > > > > > +static int mtk_fd_suspend(struct device *dev) > > > > > > +{ > > > > > > + struct mtk_fd_dev *fd_dev; > > > > > > + int ret; > > > > > > + > > > > > > + if (pm_runtime_suspended(dev)) > > > > > > + return 0; > > > > > > + > > > > > > + fd_dev = dev_get_drvdata(dev); > > > > > > + > > > > > > + if (atomic_read(&fd_dev->fd_hw.fd_user_cnt) > 0) { > > > > > > + ret = pm_runtime_put_sync(fd_dev->fd_hw.larb_dev); > > > > > > + clk_disable_unprepare(fd_dev->fd_hw.fd_clk); > > > > > > + return ret; > > > > > > + } > > > > > > > > > > This isn't going to work, because the hardware may be still processing a > > > > > frame at this point. You need a way to ensure that the hardware goes idle > > > > > here first and then in resume, you need to make the hardware continue when > > > > > it left before suspend. > > > > > > > > > For this part, I would like to do as following: > > > > when suspend, it should set the driver power state as idle or suspended > > > > to stop further enqueue jobs, should be judged in mtk_fd_hw_job_exec() > > > > or somewhere, then wait for the unfinished job return or timeout, and > > > > finally close the clock. > > > > When resume, we set the driver power state as resumed and let the new > > > > jobs to be enqueued. > > > > > > > > Or another way is to create a wait queue or work queue to store the jobs > > > > from user. When suspend, we change the driver status to restrict the new > > > > jobs joining to work queue and close the clock. When resume, driver > > > > continue execute the jobs from the work queue. > > > > > > > > > > I wouldn't introduce a workqueue only for handling suspend/resume. If > > > we end up in a need to use a workqueue for some other purposes too, > > > then a freezable workqueue could work for blocking further requests > > > during suspend indeed. If we don't need a workqueue for anything else, > > > then a simple boolean flag set and wait for last job to finish in > > > suspend and flag reset and call to schedule a next job in resume > > > should be good enough. > > > > > > Best regards, > > > Tomasz > > > > Ok, we got it. > > > > Sincerely, > > Jerry > > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel