From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6B76C07E96 for ; Wed, 7 Jul 2021 01:44:53 +0000 (UTC) Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D79C161CB2 for ; Wed, 7 Jul 2021 01:44:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D79C161CB2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 66B4482D76; Wed, 7 Jul 2021 03:44:50 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (2048-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="gqE6aXN0"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id CCB1D82D97; Wed, 7 Jul 2021 03:44:48 +0200 (CEST) Received: from mail-pf1-x436.google.com (mail-pf1-x436.google.com [IPv6:2607:f8b0:4864:20::436]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 6413882C91 for ; Wed, 7 Jul 2021 03:44:45 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=takahiro.akashi@linaro.org Received: by mail-pf1-x436.google.com with SMTP id y4so719371pfi.9 for ; Tue, 06 Jul 2021 18:44:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=sk+MUqeqTg8VMIS1KS0ByuwFpbCN0iUTcSOw8CKUz28=; b=gqE6aXN0X2CoJ19+urzQDJz5JVT8HfF3f0OcRN2ISPrbLpg32tZ5UOjfZTIinHQSSc O8Te+TOu28GZW10hi6+7HoI7n8Xipb9k66lX+06DfH47uQ5EOLy02p/I/mSmONfplljF govJlIJAlyFLdb9Xi07gJ11UC09cHGnP9Wgak+PvDNYK5WMl8RkIA2Bhj2m8dUYxb94+ 5AYtb2+MVxiBkzhTDM7bXWzjkFgMJmu9nlkDHaX6ES/W/+DfI7GhQQCt1abGRgsa+Opm 9M9V8yfek6Z+CgII+GWk8D9BOTfeh/gtvZgyRa6rAO14XYe2dE25jhzWKte1P80d9MIN qjEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id :mail-followup-to:references:mime-version:content-disposition :content-transfer-encoding:in-reply-to; bh=sk+MUqeqTg8VMIS1KS0ByuwFpbCN0iUTcSOw8CKUz28=; b=KTrOIR8b41+hy/ikKNCn44GkH6WDWFjMdmQauqfnY9j9HaARL/b1+M/YiqIvSB6ebk 8QxOnVXtNUJ3u5jfGezUTGoUS2+YQQI4PbO00StZRIRRmbR66wj45ktwj1JIGw8MjG4w cgo9Ss7JS2WALGTQNZnmHhL9uMqof299ZdQwigBdhEJAZKniFDze/4gJehVPDpZ+/acB ngkVOQH7MK0W+wDcl9POz98PCxBeJBvkqT39S/djLSMgodZ3PY5hY6msZP+nNUgfWPDW k5IK7Rpm+mW1QXV+ih1T4fhRKZ/A9XLGkSweh7JP0c7ipk79McujL3wEH1IhJCMLUnEk C2Hw== X-Gm-Message-State: AOAM530o6tiEO0yma98laFvt+Af2eGKKTrTZN1kARGspHeRef1PQODAY yml8yc8DZGIS0IWXY4jPOTNQtA== X-Google-Smtp-Source: ABdhPJw3I68XW3KmXNUtk/AlBvUQGKCVdwu9nTjCLNf6t2Q7I7Q2wrLCXEfBX04IcO4EXpAAOeUNsA== X-Received: by 2002:a63:e118:: with SMTP id z24mr23626657pgh.212.1625622283516; Tue, 06 Jul 2021 18:44:43 -0700 (PDT) Received: from laputa (p3dd30549.tkyea130.ap.so-net.ne.jp. [61.211.5.73]) by smtp.gmail.com with ESMTPSA id t1sm15468858pjo.33.2021.07.06.18.44.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Jul 2021 18:44:42 -0700 (PDT) Date: Wed, 7 Jul 2021 10:44:35 +0900 From: AKASHI Takahiro To: Heinrich Schuchardt Cc: Fran??ois Ozog , Ilias Apalodimas , Tuomas Tynkkynen , U-Boot Mailing List Subject: Re: QEMU NUMA and U-Boot Message-ID: <20210707014435.GA24369@laputa> Mail-Followup-To: AKASHI Takahiro , Heinrich Schuchardt , Fran??ois Ozog , Ilias Apalodimas , Tuomas Tynkkynen , U-Boot Mailing List References: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.2 at phobos.denx.de X-Virus-Status: Clean François, On Tue, Jul 06, 2021 at 08:10:08PM +0200, Heinrich Schuchardt wrote: > On 7/6/21 6:13 PM, François Ozog wrote: > > Hi Heinrich, U-Boot 2021-07rc5 does not take into account memory > > description when using Qemu 5.2 NUMA configuration to adapt memory map > > (kernel_addr_r...): > > > >        -smp 4 \ > >          -m 8G,slots=2,maxmem=16G \ > >         -object memory-backend-ram,size=4G,id=m0 \ > >         -object memory-backend-ram,size=4G,id=m1 \ > >         -numa node,cpus=0-1,nodeid=0,memdev=m0 \ > >         -numa node,cpus=2-3,nodeid=1,memdev=m1 > > > > kernel_addr_r is still 0x4040000 and thus you can't use it to bootefi. > > > > fdt addr 0x13ede6de0; fdt print > > > > Displays fdt while I think it should not. > > > > If I load the kernel at dram.start, the load works but not boot > > > > U-Boot 2021.07 (Jul 06 2021 - 13:26:43 +0000) > > > > > > DRAM:4 GiB > > > > Flash: 64 MiB > > > > Loading Environment from Flash... OK > > > > In:pl011@9000000 > > > > Out: pl011@9000000 > > > > Err: pl011@9000000 > > > > Net: eth0: virtio-net#32 > > > > Hit any key to stop autoboot:0 > > > > => > > > > => bdinfo > > > > boot_params = 0x0000000000000000 > > > > DRAM bank = 0x0000000000000000 > > > > -> start= 0x0000000140000000 > > > > -> size = 0x0000000100000000 > > > > flashstart= 0x0000000000000000 > > > > flashsize = 0x0000000004000000 > > > > flashoffset = 0x00000000000bc990 > > > > baudrate= 115200 bps > > > > relocaddr = 0x000000013ff27000 > > > > reloc off = 0x000000013ff27000 > > > > Build = 64-bit > > > > current eth = virtio-net#32 > > > > ethaddr = 52:52:52:52:52:52 > > > > IP addr = > > > > fdt_blob= 0x000000013ede6de0 > > > > new_fdt = 0x000000013ede6de0 > > > > fdt_size= 0x0000000000100000 > > > > lmb_dump_all: > > > > memory.cnt= 0x1 > > > > memory.reg[0x0].base = 0x140000000 > > > > .size = 0x100000000 > > > > > > reserved.cnt= 0x0 > > > > arch_number = 0x0000000000000000 > > > > TLB addr= 0x000000013fff0000 > > > > irq_sp= 0x000000013ede6dd0 > > > > sp start= 0x000000013ede6dd0 > > > > Early malloc usage: 3a8 / 2000 > > > > => load virtio 0:1 0x140000000 /oskit.efi > > > > 853424 bytes read in 1 ms (813.9 MiB/s) > > > > => bootefi0x140000000 0x13ede6dd0 > > > > ERROR: Failed to register WaitForKey event > > > > Setting OsIndications failed > > > > Error: Cannot initialize UEFI sub-system, r = 9 > > > > > > I think there is a need to calculate memory map based on previous > > firmware (TFA, QEMU can be considered as previous frimware) information > > (DT or blob_list). > > > > What do you think ? > > > > Cheers > > > > FF > > > > -- > > > > François-Frédéric Ozog | /Director Business Development/ > > T: +33.67221.6485 > > francois.ozog@linaro.org | Skype: ffozog > > > > > > The kernel load address is hard coded here: > include/configs/qemu-arm.h:41: "kernel_addr_r=0x40400000\0" \ > > bdinfo shows: > DRAM start = 0x140000000 > DRAM size = 0x100000000 > > fdt addr $fdt_addr > fdt printf > > shows two memory areas. One at 40000000, one at 140000000. (This shows that U-Boot receives a correct memory map via dtb.) Is this a NUMA machine, isn't it? Why should we care of which memory region be used here? Please note that this is a virtual machine, there is no practical difference between two regions. The root problem is that U-Boot did not recognize there were two memory regions. We can fix this issue in either way: 1) diff --git a/configs/qemu_arm64_defconfig b/configs/qemu_arm64_defconfig index f6e586627a8e..b70ffae8bf6e 100644 --- a/configs/qemu_arm64_defconfig +++ b/configs/qemu_arm64_defconfig @@ -1,7 +1,7 @@ CONFIG_ARM=y CONFIG_POSITION_INDEPENDENT=y CONFIG_ARCH_QEMU=y -CONFIG_NR_DRAM_BANKS=1 +CONFIG_NR_DRAM_BANKS=2 CONFIG_ENV_SIZE=0x40000 CONFIG_ENV_SECT_SIZE=0x40000 CONFIG_AHCI=y 2) diff --git a/lib/fdtdec.c b/lib/fdtdec.c index 4b097fb588ed..4067ea2dead6 100644 --- a/lib/fdtdec.c +++ b/lib/fdtdec.c @@ -1111,7 +1111,7 @@ int fdtdec_setup_memory_banksize(void) return -EINVAL; } - for (bank = 0; bank < CONFIG_NR_DRAM_BANKS; bank++) { + for (bank = 0; ; bank++) { ret = ofnode_read_resource(mem, reg++, &res); if (ret < 0) { reg = 0; (fdtdec_setup_memory_banksize() is called in dram_init_banksize().) (2) seems much better, but I don't know why we had to use CONFIG_NR_DRAM_BANKS here. In this case, other occurrences of CONFIG_NR_DRAM_BANKS in this file should be replaced with a variable for it. > Your use case is well beyond the typical U-Boot usage. So I guess it > will be up to Linaro to provide the necessary patches: > > * determine the active CPU > * determine the RAM assigned to the active CPU according > to the numa-node-id in the device-tree > * make sure that U-Boot only uses the memory of the active CPU > internally > * make sure that the UEFI memory map contains a compliant description > * possibly, dynamically set up the environment variables > > +CC Tuomas Tynkkynen (maintainer for qemu_arm64_defconfig) For (1), we'd better have a different config, or increase the value of CONFIG_NR_DRAM_BANKS to a bigger number? -Takahiro Akashi > Best regards > > Heinrich