From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jesper Dangaard Brouer Subject: Re: Kernel 4.19 network performance - forwarding/routing normal users traffic Date: Sat, 3 Nov 2018 13:58:31 +0100 Message-ID: <20181103135831.180628ab@redhat.com> References: <61697e49-e839-befc-8330-fc00187c48ee@itcare.pl> <61e30474-b5e9-4dc8-a8a6-90cdd17d2a66@gmail.com> <8e10bf68-f3b3-98f2-91a5-25b151756dd6@itcare.pl> <20181101102213.2fa2643d@redhat.com> <20181101152716.GA13895@intel.com> <20181102052356.GA17587@intel.com> <20181102124037.352b15de@redhat.com> <20181102142024.GA18343@intel.com> <08828f13-1e47-e65b-caa8-2319167fc495@itcare.pl> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8BIT Cc: Aaron Lu , Saeed Mahameed , "eric.dumazet@gmail.com" , "netdev@vger.kernel.org" , Tariq Toukan , "ilias.apalodimas@linaro.org" , "yoel@kviknet.dk" , "mgorman@techsingularity.net" , brouer@redhat.com To: =?UTF-8?B?UGF3ZcWC?= Staszewski Return-path: Received: from mx1.redhat.com ([209.132.183.28]:57024 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727256AbeKCWJy (ORCPT ); Sat, 3 Nov 2018 18:09:54 -0400 In-Reply-To: Sender: netdev-owner@vger.kernel.org List-ID: On Sat, 3 Nov 2018 01:16:08 +0100 Paweł Staszewski wrote: > W dniu 02.11.2018 o 20:02, Paweł Staszewski pisze: > > > > > > W dniu 02.11.2018 o 15:20, Aaron Lu pisze: > >> On Fri, Nov 02, 2018 at 12:40:37PM +0100, Jesper Dangaard Brouer wrote: > >>> On Fri, 2 Nov 2018 13:23:56 +0800 > >>> Aaron Lu wrote: > >>> > >>>> On Thu, Nov 01, 2018 at 08:23:19PM +0000, Saeed Mahameed wrote: > >>>>> On Thu, 2018-11-01 at 23:27 +0800, Aaron Lu wrote: > >>>>>> On Thu, Nov 01, 2018 at 10:22:13AM +0100, Jesper Dangaard Brouer > >>>>>> wrote: > >>>>>> ... ... [...] > >>> TL;DR: this is order-0 pages (code-walk trough proof below) > >>> > >>> To Aaron, the network stack *can* call __free_pages_ok() with order-0 > >>> pages, via: [...] > >> > >> I think here is a problem - order 0 pages are freed directly to buddy, > >> bypassing per-cpu-pages. This might be the reason lock contention > >> appeared on free path. Can someone apply below diff and see if lock > >> contention is gone? > >> > > Will test it tonight > > > Patch applied > perf report: > https://ufile.io/sytfh > > > But i need to wait also with more traffic currently cpu's are sleeping > Well, that would be the expected result, that the CPUs get more time to sleep, if the lock contention is gone... What is the measured bandwidth now? Notice, you might still be limited by the PCIe bandwidth, but then your CPUs might actually decide to sleep, as they are getting data fast enough. [...] > >> > >> diff --git a/mm/page_alloc.c b/mm/page_alloc.c > >> index e2ef1c17942f..65c0ae13215a 100644 > >> --- a/mm/page_alloc.c > >> +++ b/mm/page_alloc.c > >> @@ -4554,8 +4554,14 @@ void page_frag_free(void *addr) > >>   { > >>       struct page *page = virt_to_head_page(addr); > >>   -    if (unlikely(put_page_testzero(page))) > >> -        __free_pages_ok(page, compound_order(page)); > >> +    if (unlikely(put_page_testzero(page))) { > >> +        unsigned int order = compound_order(page); > >> + > >> +        if (order == 0) > >> +            free_unref_page(page); > >> +        else > >> +            __free_pages_ok(page, order); > >> +    } -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer