From mboxrd@z Thu Jan 1 00:00:00 1970 Received: by 2002:a17:906:779e:b0:94e:fe67:1757 with SMTP id s30csp2042003ejm; Tue, 2 May 2023 02:04:06 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4c6VmW/O5SUsnBkc+l03vwStvkHJlShY85ifqck11W6N9WzMvsW35XxvzHTGIjzK3wKMtH X-Received: by 2002:ac8:5f87:0:b0:3e6:98a5:a965 with SMTP id j7-20020ac85f87000000b003e698a5a965mr23576226qta.22.1683018245782; Tue, 02 May 2023 02:04:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683018245; cv=none; d=google.com; s=arc-20160816; b=L+JRuGQ9wEnh3JNo1cLKHTV64aKLp7AzMoipsSA8ao/HDlwgSoOR+mNLfAjemKCcDm qi+wdbJoOweln41hSinnXKEB6Ybfbu7PsOejV2K9Y4vSZZ9kspnAZnmB9ekpGd6eR+bm +nRCePIaT6RurdKvtlEZ9Q4OK6hqar/MDtoNw8IQrSDvjr77JH+Q40uYqk69fmW1/KpL sw5cQ5LGl1BCWWFpaXr8WwBRxeU4IC9IoGQWo5fgcOAnkzcl1ZZ9ExpGNzcVEDOp/IqT +D3qTq377fQ+Li0EFZjNfNoSkIhAwfrczAc8BUPkUA5VPCHMRenchoXF2+TpMXuTnGVu GLHg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mime-version:message-id:date :user-agent:references:organization:in-reply-to:subject:cc:to:from :dkim-signature; bh=cSbRx1H+cG8NzIjCF3YLAeY91SKkBCotNuy/KA1bti4=; b=vwET8J7JKthuvyEd6ou75p4Em65oO9B+H8/VBuCQEpV70Mcc+JS2x4nhs6mbJOLlFF jrGOvs2RFEx/S9eUwwArluB4D/NniXRChOcD4nQbUDpMAst0xvdefRrG0uttf9Hr03hd VBwGN/enIX9zN34QIyTkM1OsqngEAS71EZCEhiIiimSetGWS5RfrCtMG45jK8ZjYYEE1 PaTtKCz0LcJtCLGzxhMsB5vX5V72U9i1lS8XDhzpLpXOQm5/LkhQd1R6scZlO/g3IWNl N3DE/3tibuXdSSH0ZeY3WwGJF3uFd1sPxUZta7Js6VWny3NtRlaQzVXsAVh9mfX69Vwe tYUw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=K1Low4jn; spf=pass (google.com: domain of qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id x5-20020a05622a000500b003e656530eb0si17771738qtw.32.2023.05.02.02.04.05 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 02 May 2023 02:04:05 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=K1Low4jn; spf=pass (google.com: domain of qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ptlv5-0001oz-Ra; Tue, 02 May 2023 05:03:35 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ptlv4-0001ob-ID for qemu-arm@nongnu.org; Tue, 02 May 2023 05:03:34 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ptlv2-0005dZ-Uj for qemu-arm@nongnu.org; Tue, 02 May 2023 05:03:34 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1683018211; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=cSbRx1H+cG8NzIjCF3YLAeY91SKkBCotNuy/KA1bti4=; b=K1Low4jnr9rM1cYsZ/cEMNYbTZobyw2PpCQibxDVkElhV4FkeA8aooh4KOBIgP05VEkQDs TlGYHqjIG050Dq4V7IxiynWXcg0SvYl2zMgWn1qAxQVQ1pjgdM4ulzAtxMVLPtkCJk/xTF PW+arRFeaO2tXGRJ/AAh1ARNM7DN0B4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-201-KVywX8IFN7u-YYNhIIOb8w-1; Tue, 02 May 2023 05:03:27 -0400 X-MC-Unique: KVywX8IFN7u-YYNhIIOb8w-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4F573885626; Tue, 2 May 2023 09:03:27 +0000 (UTC) Received: from localhost (unknown [10.39.193.83]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D30A12166B26; Tue, 2 May 2023 09:03:26 +0000 (UTC) From: Cornelia Huck To: Richard Henderson , quintela@redhat.com Cc: Peter Maydell , Paolo Bonzini , qemu-arm@nongnu.org, qemu-devel@nongnu.org, kvm@vger.kernel.org, Eric Auger , Gavin Shan , Philippe =?utf-8?Q?Mathieu-Daud=C3=A9?= , Andrea Bolognani Subject: Re: [PATCH v7 1/1] arm/kvm: add support for MTE In-Reply-To: <64915da6-4276-1603-1454-9350a44561d8@linaro.org> Organization: Red Hat GmbH References: <20230428095533.21747-1-cohuck@redhat.com> <20230428095533.21747-2-cohuck@redhat.com> <87sfcj99rn.fsf@secure.mitica> <64915da6-4276-1603-1454-9350a44561d8@linaro.org> User-Agent: Notmuch/0.37 (https://notmuchmail.org) Date: Tue, 02 May 2023 11:03:25 +0200 Message-ID: <871qjzcdgi.fsf@redhat.com> MIME-Version: 1.0 Content-Type: text/plain X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 Received-SPF: pass client-ip=170.10.129.124; envelope-from=cohuck@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.171, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-arm@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org Sender: qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org X-TUID: QxNz43/u4uIl On Mon, May 01 2023, Richard Henderson wrote: > On 4/28/23 18:50, Juan Quintela wrote: >> Pardon my ignorance here, but to try to help with migration. How is >> this mte tag stored? >> - 1 array of 8bits per page of memory >> - 1 array of 64bits per page of memory >> - whatever >> >> Lets asume that it is 1 byte per page. For the explanation it don't >> matter, only matters that it is an array of things that are one for each >> page. > > Not that it matters, as you say, but for concreteness, 1 4-bit tag per 16 bytes, packed, > so 128 bytes per 4k page. > >> So my suggestion is just to send another array: >> >> - 1 array of page addresses >> - 1 array of page tags that correspond to the previous one >> - 1 array of pages that correspond to the previous addresses >> >> You put compatiblity marks here and there checking that you are using >> mte (and the same version) in both sides and you call that a day. > > Sounds reasonable. Yes, something like that sounds reasonable as an interface. > >> Notice that this requires the series (still not upstream but already on >> the list) that move the zero page detection to the multifd thread, >> because I am assuming that zero pages also have tags (yes, it was not a >> very impressive guess). > > Correct. "Proper" zero detection would include checking the tags as well. > Zero tags are what you get from the kernel on a new allocation. > >> Now you need to tell me if I should do this for each page, or use some >> kind of scatter-gather function that allows me to receive the mte tags >> from an array of pages. > > That is going to depend on if KVM exposes an interface with which to bulk-set tags (STGM, > "store tag multiple", is only available to kernel mode for some reason), a-la > arch/arm64/mm/copypage.c (which copies the page data then the page tags separately). > > For the moment, KVM believes that memcpy from userspace is sufficient, which means we'll > want a custom memcpy using STGP to store 16 bytes along with its tag. > >> You could pass this information when we are searching for dirty pages, >> but it is going to be complicated doing that (basically we only pass the >> dirty page id, nothing else). > > A page can be dirtied by changing nothing but a tag. > So we cannot of course send tags "early", they must come with the data. > I'm not 100% sure I understood your question here. Last time MTE migration came up, we thought that we would need to go with an uffd extension (page + extra data) to handle the postcopy case properly (i.e. use some kind of array for precopy, and that interface extension for postcopy.) TBH, I'm not sure if multifd makes any difference here. > >> Another question, if you are using MTE, all pages have MTE, right? >> Or there are other exceptions? > > No such systems are built yet, so we won't know what corner cases the host system will > have to cope with, but I believe as written so far all pages must have tags when MTE is > enabled by KVM. Has anyone been able to access a real system with MTE? (All the systems where I had hoped that MTE would be available didn't have MTE in the end so far, so I'd be interested to hear if anybody else already got to play with one.) Honestly, I don't want to even try to test migration if I only have access to MTE on the FVP...