From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40318C4332F for ; Tue, 18 Oct 2022 09:35:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B1AD56B0078; Tue, 18 Oct 2022 05:35:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ACAAC6B007B; Tue, 18 Oct 2022 05:35:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9922B8E0002; Tue, 18 Oct 2022 05:35:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 8938A6B0078 for ; Tue, 18 Oct 2022 05:35:22 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 480B580F34 for ; Tue, 18 Oct 2022 09:35:21 +0000 (UTC) X-FDA: 80033562042.06.5C9947C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf10.hostedemail.com (Postfix) with ESMTP id CA372C0021 for ; Tue, 18 Oct 2022 09:35:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1666085720; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9aL/XJR8EJ0pRD4xfXjlEdra6nq4SqKP4CXxDvLcpoE=; b=IYsjjU/t6e0BlUB8f3Wq/31rE2a32aoZVrzFcjqKPMLGH/hIgek4kN8ImY8dD4l1o43BnH YA80pJ2T0nduV7RlvUkWGjFfUyJewf92jisgPYIbhQ5DvUDzlJAyC4D3NRSLkYifgGmwop hvMf2ZHuupMz8VewKVF4jhBQYYcMHAE= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-630-AtwxXaNFOB2kjLg1xjvhDA-1; Tue, 18 Oct 2022 05:35:18 -0400 X-MC-Unique: AtwxXaNFOB2kjLg1xjvhDA-1 Received: by mail-wm1-f72.google.com with SMTP id 125-20020a1c0283000000b003c6d73209b0so2483689wmc.1 for ; Tue, 18 Oct 2022 02:35:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:organization:from:references :cc:to:content-language:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=9aL/XJR8EJ0pRD4xfXjlEdra6nq4SqKP4CXxDvLcpoE=; b=O9VcxAN8DN5rYOIC+KBJ/7013VN4Osx6GxvDjEwMMkmZmGexB+rD4NEsTAI+uLOY+i YebctjsGAxNTJ7Gr+lqsZo1BjP5lpm7YDOEfrRVjFGmSLDHpN9BaXOSsPPyTsfNd4TAa pBK1YTLoc6RYcytP/3Zuclqb4eLGMa/kJmgNLntdpkVHdTSicCt8X940ZN+7PfbkWTGe 4pjoBmviYMc/znEZmU0Eqe1oVjMVRNVimiGOmVzCeJsxHsDPOliBXY0+BiA4C7KqabIZ CBvK45zyzjR0mKtrOJtJYU7IbihTCrfPifwYvFPwuEO1uCEJkFSlIt8XPgfyOYNiBRWX MQgw== X-Gm-Message-State: ACrzQf3QCl70GI1Un3G+ZsHMP8JU7WVNwF5jZvggUaxhLwrp5nNrn8mw BwYa6gNI44GmtNYVendVWXDFVChbQm4g7TghkpUCwlFhuMxXv645/sZ8VD3nYmv1MPaTmMkTHwG CEl1vuzpqONw= X-Received: by 2002:a5d:5a8f:0:b0:22f:9396:4908 with SMTP id bp15-20020a5d5a8f000000b0022f93964908mr1270919wrb.273.1666085717555; Tue, 18 Oct 2022 02:35:17 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4UetMEzeww9mLgSKaEV1jjxk2feDBeaPeSqoLYUq6qo+4vH8HgLMrTveLBWXoVXoX0Tyi+Ng== X-Received: by 2002:a5d:5a8f:0:b0:22f:9396:4908 with SMTP id bp15-20020a5d5a8f000000b0022f93964908mr1270910wrb.273.1666085717227; Tue, 18 Oct 2022 02:35:17 -0700 (PDT) Received: from ?IPV6:2003:cb:c705:8900:d1f:5430:86b1:31ba? (p200300cbc70589000d1f543086b131ba.dip0.t-ipconnect.de. [2003:cb:c705:8900:d1f:5430:86b1:31ba]) by smtp.gmail.com with ESMTPSA id g33-20020a05600c4ca100b003bfaba19a8fsm12375896wmp.35.2022.10.18.02.35.16 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 18 Oct 2022 02:35:16 -0700 (PDT) Message-ID: Date: Tue, 18 Oct 2022 11:35:15 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.3.1 Subject: Re: [PATCH] mm: hugetlb: support get/set_policy for hugetlb_vm_ops To: =?UTF-8?B?6buE5p2w?= Cc: Muchun Song , Mike Kravetz , Andrew Morton , linux-mm@kvack.org, linux-kernel References: <20221012081526.73067-1-huangjie.albert@bytedance.com> <2aaf2c3a-6e49-abb9-b9c8-19ce87404982@redhat.com> <2f41fc4c-68eb-ab7d-970b-fcb10f474fd4@redhat.com> <03b90a2f-2854-6e19-6ccd-41f9933d8813@redhat.com> From: David Hildenbrand Organization: Red Hat In-Reply-To: X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666085720; a=rsa-sha256; cv=none; b=ABMs9w1lSkaXN6zEAwRi9z+Ar090VtihI3Dk8W4iCjU/feXsyomzOUhWr7DNgOQKYj3+J8 b8ac5jH2LmBiLMh2GElSauVWXR03YuwVzR+lBJnZL4NrYx6XpfAKfXr5eIrD6e6BMBo4AQ ErmcgHlQ8s1i759CMvS38IeHlW6pkxE= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="IYsjjU/t"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf10.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666085720; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9aL/XJR8EJ0pRD4xfXjlEdra6nq4SqKP4CXxDvLcpoE=; b=pDwwsgjQ/4j7t5SUj4U+0+y8q50Tlh9BLkaO+EVx+RkcO5ad2uPmI2QJt/+KR36uRDe95T Hv12ZoMw0uQJKy1wx+MWNpg+pFFAcuPOZeXoH0/4jcnkaLQzZ46Dlj4T+JOAnN00WoF2Zt GsSuTueCI0bEIr4S5pxOhKBZjx66qEE= X-Stat-Signature: 9n1omjq9ksa49mns57gmbb3dai78pnbq X-Rspamd-Queue-Id: CA372C0021 X-Rspam-User: Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="IYsjjU/t"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf10.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com X-Rspamd-Server: rspam11 X-HE-Tag: 1666085720-296842 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 18.10.22 11:27, 黄杰 wrote: > David Hildenbrand 于2022年10月17日周一 20:00写道： >> >> On 17.10.22 13:46, 黄杰 wrote: >>> David Hildenbrand 于2022年10月17日周一 19:33写道： >>>> >>>> On 17.10.22 11:48, 黄杰 wrote: >>>>> David Hildenbrand 于2022年10月17日周一 16:44写道： >>>>>> >>>>>> On 12.10.22 10:15, Albert Huang wrote: >>>>>>> From: "huangjie.albert" >>>>>>> >>>>>>> implement these two functions so that we can set the mempolicy to >>>>>>> the inode of the hugetlb file. This ensures that the mempolicy of >>>>>>> all processes sharing this huge page file is consistent. >>>>>>> >>>>>>> In some scenarios where huge pages are shared: >>>>>>> if we need to limit the memory usage of vm within node0, so I set qemu's >>>>>>> mempilciy bind to node0, but if there is a process (such as virtiofsd) >>>>>>> shared memory with the vm, in this case. If the page fault is triggered >>>>>>> by virtiofsd, the allocated memory may go to node1 which depends on >>>>>>> virtiofsd. >>>>>>> >>>>>> >>>>>> Any VM that uses hugetlb should be preallocating memory. For example, >>>>>> this is the expected default under QEMU when using huge pages. >>>>>> >>>>>> Once preallocation does the right thing regarding NUMA policy, there is >>>>>> no need to worry about it in other sub-processes. >>>>>> >>>>> >>>>> Hi, David >>>>> thanks for your reminder >>>>> >>>>> Yes, you are absolutely right, However, the pre-allocation mechanism >>>>> does solve this problem. >>>>> However, some scenarios do not like to use the pre-allocation mechanism, such as >>>>> scenarios that are sensitive to virtual machine startup time, or >>>>> scenarios that require >>>>> high memory utilization. The on-demand allocation mechanism may be better, >>>>> so the key point is to find a way support for shared policy。 >>>> >>>> Using hugetlb -- with a fixed pool size -- without preallocation is like >>>> playing with fire. Hugetlb reservation makes one believe that on-demand >>>> allocation is going to work, but there are various scenarios where that >>>> can go seriously wrong, and you can run out of huge pages. >>>> >>>> If you're using hugetlb as memory backend for a VM without >>>> preallocation, you really have to be very careful. I can only advise >>>> against doing that. >>>> >>>> >>>> Also: why does another process read/write *first* to a guest physical >>>> memory location before the OS running inside the VM even initialized >>>> that memory? That sounds very wrong. What am I missing? >>>> >>> >>> for example : virtio ring buffer. >>> For the avial descriptor, the guest kernel only gives an address to >>> the backend, >>> and does not actually access the memory. >> >> Okay, thanks. So we're essentially providing uninitialized memory to a >> device? Hm, that implies that the device might have access to memory >> that was previously used by someone else ... not sure how to feel about >> that, but maybe this is just the way of doing things. >> >> The "easy" user-space fix would be to simply similarly mbind() in the >> other processes where we mmap(). Has that option been explored? > > This can also solve the problem temporarily, but we need to change all > processes that share memory with it, so it can't be done once and for > all How many process that really care are we walking about? Usually, extending the vhost-user protocol and relevant libraries should be good enough. -- Thanks, David / dhildenb