From mboxrd@z Thu Jan 1 00:00:00 1970 From: p.fedin@samsung.com (Pavel Fedin) Date: Wed, 02 Dec 2015 13:31:42 +0300 Subject: [PATCH 3/6] net: thunderx: Increase transmit queue length In-Reply-To: <00cb01d12ce0$9f090540$dd1b0fc0$@samsung.com> References: <1448961223-41888-4-git-send-email-sunil.kovvuri@gmail.com> <01d101d12c46$2de43b40$89acb1c0$@samsung.com> <1448983985.25582.35.camel@edumazet-glaptop2.roam.corp.google.com> <00cb01d12ce0$9f090540$dd1b0fc0$@samsung.com> Message-ID: <00eb01d12cec$a32aa500$e97fef00$@samsung.com> To: linux-arm-kernel@lists.infradead.org List-Id: linux-arm-kernel.lists.infradead.org Hello! > > Probably you might have to set "coherent_pool" size in bootargs to a > > higher value. > > Can you please check. > > I have tried to do this. I was able to enlarge the pool up to 4MB, and still got allocation > failures. At 8MB pool preallocation stops working: > --- cut --- > Call trace: > [] __alloc_pages_nodemask+0x4f4/0x7d4 > [] atomic_pool_init+0x60/0x1a4 > [] arm64_dma_init+0x20/0x28 > [] do_one_initcall+0x8c/0x1a4 > [] kernel_init_freeable+0x154/0x1f4 > [] kernel_init+0x10/0xd8 > DMA: failed to allocate 8192 KiB pool for atomic coherent allocation > --- cut --- > and i get even worse faults in the driver. > > I know that it is possible to allocate larger pools by setting CONFIG_FORCE_MAX_ZONEORDER, > but: > a) This is done on per-platform basis. For ThunderX we used to have a patch > (http://www.spinics.net/lists/arm-kernel/msg415457.html), which never made it upstream, > because vGIC fixes stopped requiring it at some point. And also we may want to use the nicvf > driver not only on actual hardware, but also inside virtual machine in KVM. So do we need to > set CONFIG_FORCE_MAX_ZONEORDER for virt too? And what if at some point qemu emulates not only > "virt", but some other machine (let's say AMD X-Gene), and we run it on ThunderX and want to > use nicvf with this model? > b) IMHO it's not good to have a driver which simply does not work without some obscure option > in boot arguments. > > So, i see several possible ways to solve this: > > 1. Introduce some mechanism which would allow the driver to tell the kernel that it needs > coherent pool of large size. Can be problematic because the driver can be a module, and pool > allocation happens early. > 2. Can we use some other method for allocating queues, which would not require such a huge > coherent pool? > 3. The driver could check value of atomic_pool_size and adjust own memory requirements > accordingly. This indeed looks like a quick hack, but would at least make things running > quickly. I have also noticed that CONFIG_DMA_CMA is turned off in my kernel. I guess it was a leftover from old defconfig, because i carry over my .config from version to version. I enabled it and rebuilt the kernel, but in order to get the driver working with this patch i had to also add cma=32M option to kernel arguments. With default of 16M the allocation still fails. Should we add Kconfig dependencies? Kind regards, Pavel Fedin Expert Engineer Samsung Electronics Research center Russia From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755736AbbLBKbs (ORCPT ); Wed, 2 Dec 2015 05:31:48 -0500 Received: from mailout3.w1.samsung.com ([210.118.77.13]:22086 "EHLO mailout3.w1.samsung.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751416AbbLBKbq convert rfc822-to-8bit (ORCPT ); Wed, 2 Dec 2015 05:31:46 -0500 X-AuditID: cbfec7f4-f79026d00000418a-ff-565ec88f4f2c From: Pavel Fedin To: "'Sunil Kovvuri'" , "'Eric Dumazet'" Cc: "'Linux Netdev List'" , "'LKML'" , "'LAKML'" , "'Sunil Goutham'" , "'Sunil Goutham'" References: <1448961223-41888-4-git-send-email-sunil.kovvuri@gmail.com> <01d101d12c46$2de43b40$89acb1c0$@samsung.com> <1448983985.25582.35.camel@edumazet-glaptop2.roam.corp.google.com> <00cb01d12ce0$9f090540$dd1b0fc0$@samsung.com> In-reply-to: <00cb01d12ce0$9f090540$dd1b0fc0$@samsung.com> Subject: RE: [PATCH 3/6] net: thunderx: Increase transmit queue length Date: Wed, 02 Dec 2015 13:31:42 +0300 Message-id: <00eb01d12cec$a32aa500$e97fef00$@samsung.com> MIME-version: 1.0 Content-type: text/plain; charset=UTF-8 Content-transfer-encoding: 8BIT X-Mailer: Microsoft Outlook 14.0 Thread-index: AQGScnHROc58U50eD5J6ppH35rF/UQG2SVlqAT5pB6wB6Z4d1QJ5t4arAWZheWGe7vi3UA== Content-language: ru X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrPLMWRmVeSWpSXmKPExsVy+t/xK7r9J+LCDF6eFrfY9/4sm8Wmx9dY LS7vmsNmcWyBmMXlyz+YLQ583MtisajpHYsDu8eM34tYPDac6Gf12DnrLrvH5iX1Hp83yQWw RnHZpKTmZJalFunbJXBldK29zVKwVLRi27KjzA2MvYJdjJwcEgImElPXvWaBsMUkLtxbz9bF yMUhJLCUUWJK9zJmCOc7o0Tjz1XMIFVsAuoSp79+AOsQEYiQeH7gKzuIzSzwilHiWockRMNR Joknk3azgSQ4BawkHjycywRiCwu4SczZuhosziKgKrFw/VdGEJtXwFJix4ljzBC2oMSPyfdY IIaqS0yat4gZwtaWePLuAivEqQoSO86+ZoQ54ve5fVD1IhLT/t1jnsAoNAvJqFlIRs1CMmoW kpYFjCyrGEVTS5MLipPScw31ihNzi0vz0vWS83M3MUKi5csOxsXHrA4xCnAwKvHwruCJCxNi TSwrrsw9xCjBwawkwuslAxTiTUmsrEotyo8vKs1JLT7EKM3BoiTOO3fX+xAhgfTEktTs1NSC 1CKYLBMHp1QDY59HXnOkDacXl8Hf9un2DNNlfv4ULHJR3NbQcd741+9tHipXTLQ1VBfvul0/ ycy727tUqGnPqY87Gd+fKBS//+zfF/5Hn2QD5RukDrzqZ7LhWcDqcOBfGeOL33sWz5kW97yT PWhl6nVxNTOVbx+vB87v/rYwLbl31wGjfKVSsecSiTK7vx5aqcRSnJFoqMVcVJwIAFeHvEOS AgAA Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hello! > > Probably you might have to set "coherent_pool" size in bootargs to a > > higher value. > > Can you please check. > > I have tried to do this. I was able to enlarge the pool up to 4MB, and still got allocation > failures. At 8MB pool preallocation stops working: > --- cut --- > Call trace: > [] __alloc_pages_nodemask+0x4f4/0x7d4 > [] atomic_pool_init+0x60/0x1a4 > [] arm64_dma_init+0x20/0x28 > [] do_one_initcall+0x8c/0x1a4 > [] kernel_init_freeable+0x154/0x1f4 > [] kernel_init+0x10/0xd8 > DMA: failed to allocate 8192 KiB pool for atomic coherent allocation > --- cut --- > and i get even worse faults in the driver. > > I know that it is possible to allocate larger pools by setting CONFIG_FORCE_MAX_ZONEORDER, > but: > a) This is done on per-platform basis. For ThunderX we used to have a patch > (http://www.spinics.net/lists/arm-kernel/msg415457.html), which never made it upstream, > because vGIC fixes stopped requiring it at some point. And also we may want to use the nicvf > driver not only on actual hardware, but also inside virtual machine in KVM. So do we need to > set CONFIG_FORCE_MAX_ZONEORDER for virt too? And what if at some point qemu emulates not only > "virt", but some other machine (let's say AMD X-Gene), and we run it on ThunderX and want to > use nicvf with this model? > b) IMHO it's not good to have a driver which simply does not work without some obscure option > in boot arguments. > > So, i see several possible ways to solve this: > > 1. Introduce some mechanism which would allow the driver to tell the kernel that it needs > coherent pool of large size. Can be problematic because the driver can be a module, and pool > allocation happens early. > 2. Can we use some other method for allocating queues, which would not require such a huge > coherent pool? > 3. The driver could check value of atomic_pool_size and adjust own memory requirements > accordingly. This indeed looks like a quick hack, but would at least make things running > quickly. I have also noticed that CONFIG_DMA_CMA is turned off in my kernel. I guess it was a leftover from old defconfig, because i carry over my .config from version to version. I enabled it and rebuilt the kernel, but in order to get the driver working with this patch i had to also add cma=32M option to kernel arguments. With default of 16M the allocation still fails. Should we add Kconfig dependencies? Kind regards, Pavel Fedin Expert Engineer Samsung Electronics Research center Russia