From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C40793254B0; Wed, 16 Sep 2026 14:33:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789569212; cv=none; b=QrQM17LBQIIbp2+jCF+INlFh0kpQcjK1Ie6R2fG0x6non9+4OaCWj8RKg8uXY3gFtLl/1oGD+PuR6KJawYtmCKbiHfUANvE03w8eDAvif6W7/T0vLDnVYJvB5c14o+RGm8MEznZ1tHuX+UKVQw1nt0SZ7N1tmz+LcdgCKelGO+w= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789569212; c=relaxed/simple; bh=UhMCiLnG8Ugj9w6K6IXYARcIsxqAigoIYBvU7UTHP7w=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=rXia72202b5KXaRoPfA0vDtLH/hwSHVwI16fWIjDa8IypruNRRKkWwOQJZqj7yOcPnaOJQtPkHwBCqebYIlY4MtHZOR4h99sFzlqvKN9vlKzcfs8nz9JowpPOYZzMPQeD1n7d6qPrZI63eOwUnbqRIMBGsTvmvL9cn5QDX4wI0g= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=KB+vtMog; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="KB+vtMog" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7AE971F000FF; Wed, 16 Sep 2026 14:33:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789569210; bh=PFRgUz13CKX8U9AELJWTu+7kgzvYnFqa7uF4pRvkbcU=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=KB+vtMogVxZRbbWdVaraL4Bow/O7cpdrceW8JwdKjOqho7N4jMsp6eQnMKyWxuKIQ 2JMLKxvzFMQyAjfZF/Xg/6NK6cM98R35gw17eIR9bO8V5BlHaPdRR3vNTT5SiS+ow4 Nq4TgKmxUNpvdqxstKoY2Bly2Co8I8TMAFB6kfseLHZjc3suLJDrq/tet28rC/79PU st+RcejV9HVggHtq1mGITBXWNI6YITiCKd4z8BHsNWl+5DBF3f8dW9T0sVHE93sHl3 DZIj6/SPsL0paVgMnEfQTutiq72aKiWJR7RI9Xb0HUgqlzJFfoRmqnCzbGuAvt7Coi uSWXBjxNm37Zg== Date: Wed, 16 Sep 2026 15:33:15 +0100 From: "Lorenzo Stoakes (ARM)" To: Arnd Bergmann Cc: Nick Desaulniers , Kees Cook , Linus Torvalds , Nathan Chancellor , Nicolas Schier , Bill Wendling , Justin Stitt , Masahiro Yamada , Alexey Gladkov , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexandre Ghiti , Catalin Marinas , Will Deacon , Mark Rutland , Ard Biesheuvel , Ilias Apalodimas , Josh Poimboeuf , Peter Zijlstra , Miguel Ojeda , Boqun Feng , Gary Guo , =?utf-8?B?QmrDtnJu?= Roy Baron , Benno Lossin , Andreas Hindborg , Alice Ryhl , Trevor Gross , Danilo Krummrich , Daniel Almeida , Tamir Duberstein , Alexandre Courbot , Onur =?utf-8?B?w5Z6a2Fu?= , Jonathan Corbet , Randy Dunlap , "Gustavo A. R. Silva" , linux-kbuild@vger.kernel.org, linux-kernel@vger.kernel.org, llvm@lists.linux.dev, linux-riscv@lists.infradead.org, Linux-Arch , linux-arm-kernel@lists.infradead.org, linux-efi@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-doc@vger.kernel.org, Jens Axboe , linux-hardening@vger.kernel.org Subject: Re: [PATCH v2 21/21] kbuild: use pigz for gzip compression if available Message-ID: References: <20260914-build-speedup-v2-0-39817ec5db23@kernel.org> <20260914-build-speedup-v2-21-39817ec5db23@kernel.org> <202609140844.FA7E848A@keescook> <202609150955.A51C9F7A20@keescook> <90006ad1-73df-4bd0-a434-61875edd5bb9@app.fastmail.com> Precedence: bulk X-Mailing-List: linux-kbuild@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <90006ad1-73df-4bd0-a434-61875edd5bb9@app.fastmail.com> On Tue, Sep 15, 2026 at 08:02:38PM +0200, Arnd Bergmann wrote: > On Tue, Sep 15, 2026, at 19:47, Nick Desaulniers wrote: > > On Tue, Sep 15, 2026 at 10:31 AM Kees Cook wrote: > >> On Tue, Sep 15, 2026 at 03:30:43PM +0100, Lorenzo Stoakes (ARM) wrote: > >> a) > >> My main concern is the lack of respecting the -j make argument. In my > >> mind, this is a blocker, because it means a build will now _always_ spin > >> up max CPUs (not what -j has limited it to), and for CIs, shared compute > >> systems, or whatever, this violates the requested parallelism level. For > >> example, if I'm doing a long-running Coccinelle replacement in one tree > >> (which uses half the CPUs), any builds I launch I'm asking for the other > >> half of my CPUs to be used so they don't thrash my cache. > >> > >> This is the kind of "why are all the CPUs spinning up?" question I > >> helped track down with commit 51e46c7a4007 ("docs, parallelism: Rearrange > >> how jobserver reservations are made") forever ago. > > > > Parse -j from MAKEFLAGS (https://stackoverflow.com/a/76517886), > > forward it to pigz? > > That still has a potential O(n²) problem: if one builds N kernels > with a top-level jobserver on N CPUs, and they all take the same > amount of time, you end up with N pigz processes each trying to use > N cpus at the same time. Ideally we'd use the equivalent of > -flto=auto in the compiler, which respects jobserver settings, > but I don't think pigz supports this at the moment. Hmm this is a good point :) though I see Kees has made a suggestion that could solve all of these concerns! > > Arnd -- Cheers, Lorenzo From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D734AC982C3 for ; Wed, 16 Sep 2026 14:33:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=8Kj4nNPuDa1taRdaOgkXPrJIpEw3ibASOVMldhyDYa0=; b=195ugaG8PvN8Ky B7QxI23XLi4r3nd0mHXE87SYhtOXPItZ6FCFl49y797g1nSexHsGYOhUqrKxqsHLWq7qHwuRrT20o WTgC5KwFTGJcOanQamTFuVM4jZy708UFkeg8JaxAS4tMd/uH2FThDLuIfEUeAqkLcE3v2Vy2+lr8I 0fKRwX6fyWkxXZBndFohNHzGiz7DGPO7Q4suinTIqiaV3f7DCdjaG/Jg349h52YSwtp/uu1X+agPB eIeF1yUmm2bfVgOh4uFVQTt5sw8NIRPQxmt9ISeTE71lPhBzAdarC+D2uyk0s+/0nwZxnknSLfjrf 3ip6Sq3UZpuyjQO999hQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.99.1 #2 (Red Hat Linux)) id 1x6qhb-00000009QeF-3SFY; Wed, 16 Sep 2026 14:33:37 +0000 Received: from sea.source.kernel.org ([2600:3c0a:e001:78e:0:1991:8:25]) by bombadil.infradead.org with esmtps (Exim 4.99.1 #2 (Red Hat Linux)) id 1x6qhX-00000009QdV-0Pif; Wed, 16 Sep 2026 14:33:31 +0000 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) by sea.source.kernel.org (Postfix) with ESMTP id C11BD43733; Wed, 16 Sep 2026 14:33:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7AE971F000FF; Wed, 16 Sep 2026 14:33:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1789569210; bh=PFRgUz13CKX8U9AELJWTu+7kgzvYnFqa7uF4pRvkbcU=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=KB+vtMogVxZRbbWdVaraL4Bow/O7cpdrceW8JwdKjOqho7N4jMsp6eQnMKyWxuKIQ 2JMLKxvzFMQyAjfZF/Xg/6NK6cM98R35gw17eIR9bO8V5BlHaPdRR3vNTT5SiS+ow4 Nq4TgKmxUNpvdqxstKoY2Bly2Co8I8TMAFB6kfseLHZjc3suLJDrq/tet28rC/79PU st+RcejV9HVggHtq1mGITBXWNI6YITiCKd4z8BHsNWl+5DBF3f8dW9T0sVHE93sHl3 DZIj6/SPsL0paVgMnEfQTutiq72aKiWJR7RI9Xb0HUgqlzJFfoRmqnCzbGuAvt7Coi uSWXBjxNm37Zg== Date: Wed, 16 Sep 2026 15:33:15 +0100 From: "Lorenzo Stoakes (ARM)" To: Arnd Bergmann Cc: Nick Desaulniers , Kees Cook , Linus Torvalds , Nathan Chancellor , Nicolas Schier , Bill Wendling , Justin Stitt , Masahiro Yamada , Alexey Gladkov , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexandre Ghiti , Catalin Marinas , Will Deacon , Mark Rutland , Ard Biesheuvel , Ilias Apalodimas , Josh Poimboeuf , Peter Zijlstra , Miguel Ojeda , Boqun Feng , Gary Guo , =?utf-8?B?QmrDtnJu?= Roy Baron , Benno Lossin , Andreas Hindborg , Alice Ryhl , Trevor Gross , Danilo Krummrich , Daniel Almeida , Tamir Duberstein , Alexandre Courbot , Onur =?utf-8?B?w5Z6a2Fu?= , Jonathan Corbet , Randy Dunlap , "Gustavo A. R. Silva" , linux-kbuild@vger.kernel.org, linux-kernel@vger.kernel.org, llvm@lists.linux.dev, linux-riscv@lists.infradead.org, Linux-Arch , linux-arm-kernel@lists.infradead.org, linux-efi@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-doc@vger.kernel.org, Jens Axboe , linux-hardening@vger.kernel.org Subject: Re: [PATCH v2 21/21] kbuild: use pigz for gzip compression if available Message-ID: References: <20260914-build-speedup-v2-0-39817ec5db23@kernel.org> <20260914-build-speedup-v2-21-39817ec5db23@kernel.org> <202609140844.FA7E848A@keescook> <202609150955.A51C9F7A20@keescook> <90006ad1-73df-4bd0-a434-61875edd5bb9@app.fastmail.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <90006ad1-73df-4bd0-a434-61875edd5bb9@app.fastmail.com> X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org T24gVHVlLCBTZXAgMTUsIDIwMjYgYXQgMDg6MDI6MzhQTSArMDIwMCwgQXJuZCBCZXJnbWFubiB3 cm90ZToKPiBPbiBUdWUsIFNlcCAxNSwgMjAyNiwgYXQgMTk6NDcsIE5pY2sgRGVzYXVsbmllcnMg d3JvdGU6Cj4gPiBPbiBUdWUsIFNlcCAxNSwgMjAyNiBhdCAxMDozMeKAr0FNIEtlZXMgQ29vayA8 a2Vlc0BrZXJuZWwub3JnPiB3cm90ZToKPiA+PiBPbiBUdWUsIFNlcCAxNSwgMjAyNiBhdCAwMzoz MDo0M1BNICswMTAwLCBMb3JlbnpvIFN0b2FrZXMgKEFSTSkgd3JvdGU6Cj4gPj4gYSkKPiA+PiBN eSBtYWluIGNvbmNlcm4gaXMgdGhlIGxhY2sgb2YgcmVzcGVjdGluZyB0aGUgLWogbWFrZSBhcmd1 bWVudC4gSW4gbXkKPiA+PiBtaW5kLCB0aGlzIGlzIGEgYmxvY2tlciwgYmVjYXVzZSBpdCBtZWFu cyBhIGJ1aWxkIHdpbGwgbm93IF9hbHdheXNfIHNwaW4KPiA+PiB1cCBtYXggQ1BVcyAobm90IHdo YXQgLWogaGFzIGxpbWl0ZWQgaXQgdG8pLCBhbmQgZm9yIENJcywgc2hhcmVkIGNvbXB1dGUKPiA+ PiBzeXN0ZW1zLCBvciB3aGF0ZXZlciwgdGhpcyB2aW9sYXRlcyB0aGUgcmVxdWVzdGVkIHBhcmFs bGVsaXNtIGxldmVsLiBGb3IKPiA+PiBleGFtcGxlLCBpZiBJJ20gZG9pbmcgYSBsb25nLXJ1bm5p bmcgQ29jY2luZWxsZSByZXBsYWNlbWVudCBpbiBvbmUgdHJlZQo+ID4+ICh3aGljaCB1c2VzIGhh bGYgdGhlIENQVXMpLCBhbnkgYnVpbGRzIEkgbGF1bmNoIEknbSBhc2tpbmcgZm9yIHRoZSBvdGhl cgo+ID4+IGhhbGYgb2YgbXkgQ1BVcyB0byBiZSB1c2VkIHNvIHRoZXkgZG9uJ3QgdGhyYXNoIG15 IGNhY2hlLgo+ID4+Cj4gPj4gVGhpcyBpcyB0aGUga2luZCBvZiAid2h5IGFyZSBhbGwgdGhlIENQ VXMgc3Bpbm5pbmcgdXA/IiBxdWVzdGlvbiBJCj4gPj4gaGVscGVkIHRyYWNrIGRvd24gd2l0aCBj b21taXQgNTFlNDZjN2E0MDA3ICgiZG9jcywgcGFyYWxsZWxpc206IFJlYXJyYW5nZQo+ID4+IGhv dyBqb2JzZXJ2ZXIgcmVzZXJ2YXRpb25zIGFyZSBtYWRlIikgZm9yZXZlciBhZ28uCj4gPgo+ID4g UGFyc2UgLWogZnJvbSBNQUtFRkxBR1MgKGh0dHBzOi8vc3RhY2tvdmVyZmxvdy5jb20vYS83NjUx Nzg4NiksCj4gPiBmb3J3YXJkIGl0IHRvIHBpZ3o/Cj4KPiBUaGF0IHN0aWxsIGhhcyBhIHBvdGVu dGlhbCBPKG7CsikgcHJvYmxlbTogaWYgb25lIGJ1aWxkcyBOIGtlcm5lbHMKPiB3aXRoIGEgdG9w LWxldmVsIGpvYnNlcnZlciBvbiBOIENQVXMsIGFuZCB0aGV5IGFsbCB0YWtlIHRoZSBzYW1lCj4g YW1vdW50IG9mIHRpbWUsIHlvdSBlbmQgdXAgd2l0aCBOIHBpZ3ogcHJvY2Vzc2VzIGVhY2ggdHJ5 aW5nIHRvIHVzZQo+IE4gY3B1cyBhdCB0aGUgc2FtZSB0aW1lLiBJZGVhbGx5IHdlJ2QgdXNlIHRo ZSBlcXVpdmFsZW50IG9mCj4gLWZsdG89YXV0byBpbiB0aGUgY29tcGlsZXIsIHdoaWNoIHJlc3Bl Y3RzIGpvYnNlcnZlciBzZXR0aW5ncywKPiBidXQgSSBkb24ndCB0aGluayBwaWd6IHN1cHBvcnRz IHRoaXMgYXQgdGhlIG1vbWVudC4KCkhtbSB0aGlzIGlzIGEgZ29vZCBwb2ludCA6KSB0aG91Z2gg SSBzZWUgS2VlcyBoYXMgbWFkZSBhIHN1Z2dlc3Rpb24gdGhhdApjb3VsZCBzb2x2ZSBhbGwgb2Yg dGhlc2UgY29uY2VybnMhCgo+Cj4gICAgICAgQXJuZAoKLS0KQ2hlZXJzLCBMb3JlbnpvCgpfX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpsaW51eC1yaXNjdiBt YWlsaW5nIGxpc3QKbGludXgtcmlzY3ZAbGlzdHMuaW5mcmFkZWFkLm9yZwpodHRwOi8vbGlzdHMu aW5mcmFkZWFkLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2xpbnV4LXJpc2N2Cg==