From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 883A4C001DC for ; Thu, 27 Jul 2023 16:43:42 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 0F5E610E5C3; Thu, 27 Jul 2023 16:43:42 +0000 (UTC) Received: from mgamail.intel.com (unknown [134.134.136.65]) by gabe.freedesktop.org (Postfix) with ESMTPS id EA6BF10E5C3; Thu, 27 Jul 2023 16:43:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690476221; x=1722012221; h=message-id:date:mime-version:subject:to:cc:references: from:in-reply-to:content-transfer-encoding; bh=EPPEG5IjRpK3UjK14v1TE0oHNl+auGdkynzlUa5wKQI=; b=KCLBJPEDMythTfgwAbVvrBqE7BYTpkiJ5kaVTI/RQoDiZLc4jp/u0wgx mT7OcAC+oFwR6LdNnMAgWAUOiGKyF/ii+MWpaUfn9YQsbUGJ0TWEuiSgO uFBhCSIh21V45t8BW/0F6W41cjmR5/Kn8jBUIz9LuxMiFJ/dZQzAairvj 2H9SkhTM/XvBbJAcD/U5XwMn47ISa/o15CvKG+nqO9+jylpIF++Jg4BTX e2Q2JxQwATPbqAmyYM5kzUY+/iYRDWaKlYrtFNRMBSzzHVpYDKg3bVoCx gMu5hPhFK89sHshI6OfLOsA+1W6S4nXvel+wPFgMaKGhPoSvkUg8oAlOV w==; X-IronPort-AV: E=McAfee;i="6600,9927,10784"; a="371996233" X-IronPort-AV: E=Sophos;i="6.01,235,1684825200"; d="scan'208";a="371996233" Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Jul 2023 09:43:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10784"; a="730416549" X-IronPort-AV: E=Sophos;i="6.01,235,1684825200"; d="scan'208";a="730416549" Received: from jlenehan-mobl1.ger.corp.intel.com (HELO [10.213.228.208]) ([10.213.228.208]) by fmsmga007-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Jul 2023 09:43:36 -0700 Message-ID: <6ed5478c-8a67-5ad8-3e6d-51c300702cf3@linux.intel.com> Date: Thu, 27 Jul 2023 17:43:33 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 Content-Language: en-US To: Maarten Lankhorst , Tejun Heo References: <20230712114605.519432-1-tvrtko.ursulin@linux.intel.com> <20230712114605.519432-17-tvrtko.ursulin@linux.intel.com> <05178cf3-df1c-80a7-12ad-816fafbc2df7@linux.intel.com> From: Tvrtko Ursulin Organization: Intel Corporation UK Plc In-Reply-To: <05178cf3-df1c-80a7-12ad-816fafbc2df7@linux.intel.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Subject: Re: [Intel-gfx] [PATCH 16/17] cgroup/drm: Expose memory stats X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Rob Clark , Kenny.Ho@amd.com, Dave Airlie , =?UTF-8?Q?St=c3=a9phane_Marchesin?= , Daniel Vetter , Intel-gfx@lists.freedesktop.org, linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org, =?UTF-8?Q?Christian_K=c3=b6nig?= , Zefan Li , Johannes Weiner , cgroups@vger.kernel.org, Eero Tamminen , "T . J . Mercier" Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" On 27/07/2023 14:42, Maarten Lankhorst wrote: > On 2023-07-26 21:44, Tejun Heo wrote: >> Hello, >> >> On Wed, Jul 26, 2023 at 12:14:24PM +0200, Maarten Lankhorst wrote: >>>> So, yeah, if you want to add memory controls, we better think >>>> through how >>>> the fd ownership migration should work. >>> >>> I've taken a look at the series, since I have been working on cgroup >>> memory >>> eviction. >>> >>> The scheduling stuff will work for i915, since it has a purely software >>> execlist scheduler, but I don't think it will work for GuC (firmware) >>> scheduling or other drivers that use the generic drm scheduler. >>> >>> For something like this,  you would probably want it to work inside >>> the drm >>> scheduler first. Presumably, this can be done by setting a weight on >>> each >>> runqueue, and perhaps adding a callback to update one for a running >>> queue. >>> Calculating the weights hierarchically might be fun.. >> >> I don't have any idea on this front. The basic idea of making high level >> distribution decisions in core code and letting individual drivers >> enforce >> that in a way which fits them the best makes sense to me but I don't know >> enough to have an opinion here. >> >>> I have taken a look at how the rest of cgroup controllers change >>> ownership >>> when moved to a different cgroup, and the answer was: not at all. If we >> >> For persistent resources, that's the general rule. Whoever instantiates a >> resource gets to own it until the resource gets freed. There is an >> exception >> with the pid controller and there are discussions around whether we want >> some sort of migration behavior with memcg but yes by and large >> instantiator >> being the owner is the general model cgroup follows. >> >>> attempt to create the scheduler controls only on the first time the >>> fd is >>> used, you could probably get rid of all the tracking. >>> This can be done very easily with the drm scheduler. >>> >>> WRT memory, I think the consensus is to track system memory like normal >>> memory. Stolen memory doesn't need to be tracked. It's kernel only >>> memory, >>> used for internal bookkeeping  only. >>> >>> The only time userspace can directly manipulate stolen memory, is by >>> mapping >>> the pinned initial framebuffer to its own address space. The only >>> allocation >>> it can do is when a framebuffer is displayed, and framebuffer >>> compression >>> creates some stolen memory. Userspace is not >>> aware of this though, and has no way to manipulate those contents. >> >> So, my dumb understanding: >> >> * Ownership of an fd can be established on the first ioctl call and >> doesn't >>    need to be migrated afterwards. There are no persistent resources to >>    migration on the first call. Yes, keyword is "can". Trouble is migration may or may not happen. One may choose "Plasma X.org" session type in your login manager and all DRM fds would be under Xorg if not migrated. Or one may choose "Plasma Wayland" and migration wouldn't matter. But former is I think has a huge deployed base so that not supporting implicit migration would be a significant asterisk next to the controller. >> * Memory then can be tracked in a similar way to memcg. Memory gets >> charged >>    to the initial instantiator and doesn't need to be moved around >>    afterwards. There may be some discrepancies around stolen memory >> but the >>    magnitude of inaccuracy introduced that way is limited and bound >> and can >>    be safely ignored. >> >> Is that correct? > > Hey, > > Yeah mostly, I think we can stop tracking stolen memory. I stopped doing > that for Xe, there is literally nothing to control for userspace in there. Right, but for reporting stolen is a red-herring. In this RFC I simply report on all memory regions known by the driver. As I said in the other reply, imagine the keys are 'system' and 'vram0'. Point was just to illustrate multiplicity of regions. Regards, Tvrtko