From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ilya Matveychikov Subject: Re: A (possible) problem with `--no-huge` option Date: Fri, 9 Jun 2017 16:08:24 +0400 Message-ID: <80B1FDDC-4B0B-4B12-843F-B3AB1587CD08@gmail.com> References: <1A9D36CE-8B6D-43A7-BE0C-9F232DFDA263@gmail.com> <20170609102727.0eb7f39d@platinum> Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: dev@dpdk.org, "adrien.mazarguil@6wind.com" , Jan Blunck , Sergio Gonzalez Monroy To: Olivier Matz Return-path: Received: from mail-wr0-f196.google.com (mail-wr0-f196.google.com [209.85.128.196]) by dpdk.org (Postfix) with ESMTP id 998AA2BB9 for ; Fri, 9 Jun 2017 14:08:29 +0200 (CEST) Received: by mail-wr0-f196.google.com with SMTP id v104so6380199wrb.0 for ; Fri, 09 Jun 2017 05:08:29 -0700 (PDT) In-Reply-To: <20170609102727.0eb7f39d@platinum> List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi Olivier, The patch from you solves the problem for me. Thank you. > On Jun 9, 2017, at 12:27 PM, Olivier Matz = wrote: >=20 > Hi Ilya, >=20 > On Sun, 14 May 2017 14:34:14 +0400, Ilya Matveychikov = wrote: >> Hi guys, >>=20 >> I have a problem while running DPDK with `--no-huge` option. It seems = that the problem occurs since commit = cdc242f260e766bd95a658b5e0686a62ec04f5b0 and that is the change that = affects me: >>=20 >> + if ((page & 0x7fffffffffffffULL) =3D=3D 0) >> + return RTE_BAD_PHYS_ADDR; >> + >>=20 >> What I did is to try to create memory pool using = rte_pktmbuf_pool_create(). I dig into the issue and found that in my = case =E2=80=9Cpage" value is 0x0080000000000000 which means that the = page is not present and =E2=80=9Csoft-dirty=E2=80=9D (according to = kernel=E2=80=99s documentation): >>=20 >> * Bits 0-54 page frame number (PFN) if present >> * Bits 0-4 swap type if swapped >> * Bits 5-54 swap offset if swapped >> * Bit 55 pte is soft-dirty (see = Documentation/vm/soft-dirty.txt) >> * Bit 56 page exclusively mapped (since 4.2) >> * Bits 57-60 zero >> * Bit 61 page is file-page or shared-anon (since 3.5) >> * Bit 62 page swapped >> * Bit 63 page present >>=20 >> So, before the change mentioned all =E2=80=9Cworks=E2=80=9D fine and = such pages were not handled. But now the check causes = rte_mempool_populate_default to fail with -EINVAL... >> Can anyone familiar with the memory pool allocation helps with the = issue? >>=20 >> Thanks in advice, >> Ilya Matveychikov. >>=20 >=20 > I can reproduce the issue: >=20 > make config T=3Dx86_64-native-linuxapp-gcc > make -j32 EXTRA_CFLAGS=3D"-O0 -g" > mkdir -p /mnt/huge > mount -t hugetlbfs nodev /mnt/huge > echo 256 > = /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages >=20 > # ok > ./build/app/testpmd -l 2,4 --log-level 8 --vdev=3Deth_null0 -- = --no-numa --total-num-mbufs=3D4096 -i --port-topology=3Dchained >=20 > # fail > ./build/app/testpmd --no-huge -l 2,4 --log-level 8 --vdev=3Deth_null0 = -- --no-numa --total-num-mbufs=3D4096 -i --port-topology=3Dchained >=20 >=20 > I confirm that rte_mem_virt2phy() returns RTE_BAD_PHYS_ADDR, > which makes rte_mempool_populate_virt() to fail. >=20 > Reverting cdc242f260e7 ("eal/linux: support running as unprivileged = user") > fixes the problem. Actually, it makes rte_mem_virt2phy() return 0 = instead > of RTE_BAD_PHYS_ADDR, which is seen as a valid address. >=20 > I think querying the physical address when using --no-huge does not = make > sense because the memory is not locked, and could be swapped. >=20 > Another strange thing, when using --no-huge, the physical address = returned > when allocating a memzone is the virtual address. >=20 > I see several solutions to fix the issue: >=20 > 1/ Always set physical addresses to RTE_BAD_PHYS_ADDR when started > with --no-huge. We consider that the physical address is invalid > in that case and must not be used. >=20 > This impacts rte_mem_virt2phy() and memzone_reserve*() functions. >=20 > In rte_mempool_populate_virt(), don't expect a physical address > if the application is started with --no-huge. >=20 > 2/ Change rte_mem_virt2phy() to return the virtual address when we > ask for the physical address when started with --no-huge. This is > wrong, but consistent with what is done in memzones today. >=20 > In rte_mem_virt2phy(), add at the beginning: >=20 > if (!rte_eal_has_hugepages()) > return (intptr_t)virtaddr; >=20 > 3/ lock pages in memory by reverting > 729f17a932dd ("mem: revert page locking when not using hugepages") >=20 > This would make the physical address available. > As explained in the commit log, this would also break the ability to > start dpdk with --no-huge for non-root users. >=20 >=20 > I think 1/ is better. I'm sending a patch in reply to this mail. > Ilya, please let me know if it fixes your issue. >=20 > Regards, > Olivier