From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.3 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36824C43461 for ; Wed, 16 Sep 2020 03:39:52 +0000 (UTC) Received: from lists.sourceforge.net (lists.sourceforge.net [216.105.38.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B513B221E5; Wed, 16 Sep 2020 03:39:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=sourceforge.net header.i=@sourceforge.net header.b="AMB1SObw"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=sf.net header.i=@sf.net header.b="Q2UTMGFz"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="UZOYnGV9" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B513B221E5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=linux-f2fs-devel-bounces@lists.sourceforge.net Received: from [127.0.0.1] (helo=sfs-ml-1.v29.lw.sourceforge.com) by sfs-ml-1.v29.lw.sourceforge.com with esmtp (Exim 4.90_1) (envelope-from ) id 1kIOIQ-00064j-3b; Wed, 16 Sep 2020 03:39:50 +0000 Received: from [172.30.20.202] (helo=mx.sourceforge.net) by sfs-ml-1.v29.lw.sourceforge.com with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kIOIN-00064U-Gu; Wed, 16 Sep 2020 03:39:47 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sourceforge.net; s=x; h=Content-Transfer-Encoding:MIME-Version:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:In-Reply-To:References:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=/dXCQImWEMRMgy6ij/p3va9TB2IqETDFho83iRHLHbo=; b=AMB1SObw9cUi5qfkYPLAvhITyP DaNfAZgDK81DO+KlMMgNoy7mLCbWPpm7I3S7NeAQuKbvU7KVPDaElfzmh9eEB2//0wjEVRlWSusm+ rA+kPWcZoM64PCgAjPpUx3Yak+/FAnJ44kiiOigdQyZxOpTaHbCxekyQIRzJKIiav8a0=; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sf.net; s=x ; h=Content-Transfer-Encoding:MIME-Version:Message-Id:Date:Subject:Cc:To:From :Sender:Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Id:List-Help:List-Unsubscribe:List-Subscribe:List-Post: List-Owner:List-Archive; bh=/dXCQImWEMRMgy6ij/p3va9TB2IqETDFho83iRHLHbo=; b=Q 2UTMGFzxxPPK+ttCvu9fB5ZY4/uN6lPG4R6LOJE7bp+s//GYwZRl5kf7gxaNqdwbGlAiXKY9uW1pt LGjqBbYmFWUn4TQcfoe5RHWBz/1/5xrmX2mcklvdzsWXacFXw6Zc7Ixukqtg+3mnLH6ur22bQkLuh aQzEFV8hnAl1QHSk=; Received: from mail-pf1-f194.google.com ([209.85.210.194]) by sfi-mx-3.v28.lw.sourceforge.com with esmtps (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.92.2) id 1kIOIK-00AlS9-HO; Wed, 16 Sep 2020 03:39:47 +0000 Received: by mail-pf1-f194.google.com with SMTP id o20so3134597pfp.11; Tue, 15 Sep 2020 20:39:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=/dXCQImWEMRMgy6ij/p3va9TB2IqETDFho83iRHLHbo=; b=UZOYnGV9xEd3Q38l0h4F3wgESyOQDPdKJ/OVp6mgk5l4DbSiGm0reMbWYRC9iCMEgA nEDVFZ89KHyLCmtzuDdqEEAHZcSRNrkllDocarL1wD/E/b5585mIw7Anp6ZmCNPCtvLh p1P9yiPMEaj0AN5iK+WWe6V3WxtnVFRKSXRggeuTWNse0wLdHWd5M6h9Srcza06DZUgp pYtYWvisuq2ox6NDtLh2GDbuuYBz55godHyZGylRMbEs+75/AA6yVB8dkFQIV9BAlFMx tSQtEpp65QtP5aNlDnXT/OsqGzJenUQa0+wIHbWvZLnbbA3RhIuT29sTajetvStVdDcb WF0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=/dXCQImWEMRMgy6ij/p3va9TB2IqETDFho83iRHLHbo=; b=iOZgWYcWAAVFVV+Ll1hQjbpFATFdL7RoZdEnP4b2fptYVViduVNqFZUwzi7uA73+cv sPUO3S90NdGGhnZjekLYphXTgukahEaDE5dw1fr8vu2qOn2gP11KRcjOAmXz4Vir603v bdNc8CezV9WMIWdU5uFxrJlxuyMkxX5ZENmSLq3hk22qsKqB9xNX+HCgC53ZSZSNd6G6 mI57MDNqYIy3O2fLy2RSqOb9RI9bX/rT3nBBK8JETfgQYxXyKXyvkE62a0PI137rV7cB VFWt2UKkuZbmslt5FTqjKx7JVhMQk29yQ1LW/GOdZA2+igexQCNPn7lmvPjL9415RmuO dfJg== X-Gm-Message-State: AOAM531vSx+uVN4+ISuSjPSYgsBZrMlUfIbSfCsR4/YmyJj58cSVDbvP /D3GiFU+IxGc6qF7udmLN/4= X-Google-Smtp-Source: ABdhPJwbCF71/tvgaLc7zOdfm6OAAXs/Xq3SpcZwyMmnkNrToq/fk1pMASBxsfBaKkP79ZPSaCBVLw== X-Received: by 2002:a63:185a:: with SMTP id 26mr224880pgy.408.1600227568286; Tue, 15 Sep 2020 20:39:28 -0700 (PDT) Received: from nickserv.localdomain (c-98-33-101-203.hsd1.ca.comcast.net. [98.33.101.203]) by smtp.gmail.com with ESMTPSA id i20sm12856635pgk.77.2020.09.15.20.39.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Sep 2020 20:39:27 -0700 (PDT) From: Nick Terrell To: Herbert Xu Date: Tue, 15 Sep 2020 20:42:53 -0700 Message-Id: <20200916034307.2092020-1-nickrterrell@gmail.com> X-Mailer: git-send-email 2.28.0 MIME-Version: 1.0 X-Headers-End: 1kIOIK-00AlS9-HO Subject: [f2fs-dev] [PATCH 0/9] Update to zstd-1.4.6 X-BeenThere: linux-f2fs-devel@lists.sourceforge.net X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: squashfs-devel@lists.sourceforge.net, Johannes Weiner , Nick Terrell , Yann Collet , linux-kernel@vger.kernel.org, linux-f2fs-devel@lists.sourceforge.net, Petr Malat , Chris Mason , Nick Terrell , linux-crypto@vger.kernel.org, Kernel Team , Niket Agarwal , linux-btrfs@vger.kernel.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: linux-f2fs-devel-bounces@lists.sourceforge.net From: Nick Terrell This patchset upgrades the zstd library to the latest upstream release. The current zstd version in the kernel is a modified version of upstream zstd-1.3.1. At the time it was integrated, zstd wasn't ready to be used in the kernel as-is. But, it is now possible to use upstream zstd directly in the kernel. I have not yet release zstd-1.4.6 upstream. I want the zstd version in the kernel to match up with a known upstream release, so we know exactly what code is running. Whenever this patchset is ready for merge, I will cut a release at the upstream commit that gets merged. This should not be necessary for future releases. The kernel zstd library is automatically generated from upstream zstd. A script makes the necessary changes and imports it into the kernel. The changes are: 1. Replace all libc dependencies with kernel replacements and rewrite includes. 2. Remove unncessary portability macros like: #if defined(_MSC_VER). 3. Use the kernel xxhash instead of bundling it. This automation gets tested every commit by upstream's continuous integration. When we cut a new zstd release, we will submit a patch to the kernel to update the zstd version in the kernel. I've updated zstd to upstream with one big patch because every commit must build, so that precludes partial updates. Since the commit is 100% generated, I hope the review burden is lightened. I considered replaying upstream commits, but that is not possible because there have been ~3500 upstream commits since the last zstd import, and the commits don't all build individually. The bulk update preserves bisectablity because bugs can be bisected to the zstd version update. At that point the update can be reverted, and we can work with upstream to find and fix the bug. After this big switch in how the kernel consumes zstd, future patches will be smaller, because they will only have one upstream release worth of changes each. This patchset comes in 3 parts: 1. The first 2 patches prepare for the zstd upgrade. The first patch adds a compatibility wrapper so zstd can be upgraded without modifying any callers. The second patch adds an indirection for the lib/decompress_unzstd.c including of all decompression source files. 2. Import zstd-1.4.6. This patch is completely generated from upstream using automated tooling. 3. Update all callers to the zstd-1.4.6 API then delete the compatibility wrapper. I tested every caller of zstd on x86_64. I tested both after the 1.4.6 upgrade using the compatibility wrapper, and after the final patch in this series. I had problems with F2FS, where I had file truncation both before and after this series, so I would appreciate help testing it. All other callers were good. I tested kernel and initramfs decompression in i386 and arm. I ran benchmarks to compare the current zstd in the kernel with zstd-1.4.6. I benchmarked on x86_64 using QEMU with KVM enabled on an Intel i9-9900k. I found: * BtrFS zstd compression at levels 1 and 3 is 5% faster * BtrFS zstd decompression+read is 15% faster * SquashFS zstd decompression+read is 15% faster * ZRAM decompression+read is 30% faster * Kernel zstd decompression is 35% faster * Initramfs zstd decompression+build is 5% faster The latest zstd also offers bug fixes and a 1 KB reduction in stack uage during compression. Please let me know if there is anything that I can do to ease the way for these patches. I think it is important because it gets large performance improvements, contains bug fixes, and is switching to a more maintainable model of consuming upstream zstd directly, making it easy to keep up to date. Best, Nick Terrell Nick Terrell (9): lib: zstd: Add zstd compatibility wrapper lib: zstd: Add decompress_sources.h for decompress_unzstd lib: zstd: Upgrade to latest upstream zstd version 1.4.6 crypto: zstd: Switch to zstd-1.4.6 API btrfs: zstd: Switch to the zstd-1.4.6 API f2fs: zstd: Switch to the zstd-1.4.6 API squashfs: zstd: Switch to the zstd-1.4.6 API lib: unzstd: Switch to the zstd-1.4.6 API lib: zstd: Remove zstd compatibility wrapper crypto/zstd.c | 22 +- fs/btrfs/zstd.c | 46 +- fs/f2fs/compress.c | 100 +- fs/squashfs/zstd_wrapper.c | 7 +- include/linux/zstd.h | 3019 ++++++++---- include/linux/zstd_errors.h | 76 + lib/decompress_unzstd.c | 44 +- lib/zstd/Makefile | 35 +- lib/zstd/bitstream.h | 379 -- lib/zstd/common/bitstream.h | 437 ++ lib/zstd/common/compiler.h | 134 + lib/zstd/common/cpu.h | 194 + lib/zstd/common/debug.c | 24 + lib/zstd/common/debug.h | 101 + lib/zstd/common/entropy_common.c | 355 ++ lib/zstd/common/error_private.c | 55 + lib/zstd/common/error_private.h | 66 + lib/zstd/common/fse.h | 709 +++ lib/zstd/common/fse_decompress.c | 380 ++ lib/zstd/common/huf.h | 352 ++ lib/zstd/common/mem.h | 347 ++ lib/zstd/common/zstd_common.c | 83 + lib/zstd/common/zstd_deps.h | 134 + lib/zstd/common/zstd_internal.h | 424 ++ lib/zstd/compress.c | 3485 -------------- lib/zstd/compress/fse_compress.c | 625 +++ lib/zstd/compress/hist.c | 165 + lib/zstd/compress/hist.h | 75 + lib/zstd/compress/huf_compress.c | 764 +++ lib/zstd/compress/zstd_compress.c | 4160 +++++++++++++++++ lib/zstd/compress/zstd_compress_internal.h | 1103 +++++ lib/zstd/compress/zstd_compress_literals.c | 158 + lib/zstd/compress/zstd_compress_literals.h | 29 + lib/zstd/compress/zstd_compress_sequences.c | 433 ++ lib/zstd/compress/zstd_compress_sequences.h | 54 + lib/zstd/compress/zstd_compress_superblock.c | 849 ++++ lib/zstd/compress/zstd_compress_superblock.h | 32 + lib/zstd/compress/zstd_cwksp.h | 524 +++ lib/zstd/compress/zstd_double_fast.c | 521 +++ lib/zstd/compress/zstd_double_fast.h | 32 + lib/zstd/compress/zstd_fast.c | 496 ++ lib/zstd/compress/zstd_fast.h | 31 + lib/zstd/compress/zstd_lazy.c | 1138 +++++ lib/zstd/compress/zstd_lazy.h | 61 + lib/zstd/compress/zstd_ldm.c | 619 +++ lib/zstd/compress/zstd_ldm.h | 104 + lib/zstd/compress/zstd_opt.c | 1200 +++++ lib/zstd/compress/zstd_opt.h | 50 + lib/zstd/decompress.c | 2531 ---------- lib/zstd/decompress/huf_decompress.c | 1205 +++++ lib/zstd/decompress/zstd_ddict.c | 241 + lib/zstd/decompress/zstd_ddict.h | 44 + lib/zstd/decompress/zstd_decompress.c | 1836 ++++++++ lib/zstd/decompress/zstd_decompress_block.c | 1540 ++++++ lib/zstd/decompress/zstd_decompress_block.h | 62 + .../decompress/zstd_decompress_internal.h | 195 + lib/zstd/decompress_sources.h | 18 + lib/zstd/entropy_common.c | 243 - lib/zstd/error_private.h | 53 - lib/zstd/fse.h | 575 --- lib/zstd/fse_compress.c | 795 ---- lib/zstd/fse_decompress.c | 325 -- lib/zstd/huf.h | 212 - lib/zstd/huf_compress.c | 772 --- lib/zstd/huf_decompress.c | 960 ---- lib/zstd/mem.h | 151 - lib/zstd/zstd_common.c | 75 - lib/zstd/zstd_compress_module.c | 79 + lib/zstd/zstd_decompress_module.c | 79 + lib/zstd/zstd_internal.h | 273 -- lib/zstd/zstd_opt.h | 1014 ---- 71 files changed, 24497 insertions(+), 13012 deletions(-) create mode 100644 include/linux/zstd_errors.h delete mode 100644 lib/zstd/bitstream.h create mode 100644 lib/zstd/common/bitstream.h create mode 100644 lib/zstd/common/compiler.h create mode 100644 lib/zstd/common/cpu.h create mode 100644 lib/zstd/common/debug.c create mode 100644 lib/zstd/common/debug.h create mode 100644 lib/zstd/common/entropy_common.c create mode 100644 lib/zstd/common/error_private.c create mode 100644 lib/zstd/common/error_private.h create mode 100644 lib/zstd/common/fse.h create mode 100644 lib/zstd/common/fse_decompress.c create mode 100644 lib/zstd/common/huf.h create mode 100644 lib/zstd/common/mem.h create mode 100644 lib/zstd/common/zstd_common.c create mode 100644 lib/zstd/common/zstd_deps.h create mode 100644 lib/zstd/common/zstd_internal.h delete mode 100644 lib/zstd/compress.c create mode 100644 lib/zstd/compress/fse_compress.c create mode 100644 lib/zstd/compress/hist.c create mode 100644 lib/zstd/compress/hist.h create mode 100644 lib/zstd/compress/huf_compress.c create mode 100644 lib/zstd/compress/zstd_compress.c create mode 100644 lib/zstd/compress/zstd_compress_internal.h create mode 100644 lib/zstd/compress/zstd_compress_literals.c create mode 100644 lib/zstd/compress/zstd_compress_literals.h create mode 100644 lib/zstd/compress/zstd_compress_sequences.c create mode 100644 lib/zstd/compress/zstd_compress_sequences.h create mode 100644 lib/zstd/compress/zstd_compress_superblock.c create mode 100644 lib/zstd/compress/zstd_compress_superblock.h create mode 100644 lib/zstd/compress/zstd_cwksp.h create mode 100644 lib/zstd/compress/zstd_double_fast.c create mode 100644 lib/zstd/compress/zstd_double_fast.h create mode 100644 lib/zstd/compress/zstd_fast.c create mode 100644 lib/zstd/compress/zstd_fast.h create mode 100644 lib/zstd/compress/zstd_lazy.c create mode 100644 lib/zstd/compress/zstd_lazy.h create mode 100644 lib/zstd/compress/zstd_ldm.c create mode 100644 lib/zstd/compress/zstd_ldm.h create mode 100644 lib/zstd/compress/zstd_opt.c create mode 100644 lib/zstd/compress/zstd_opt.h delete mode 100644 lib/zstd/decompress.c create mode 100644 lib/zstd/decompress/huf_decompress.c create mode 100644 lib/zstd/decompress/zstd_ddict.c create mode 100644 lib/zstd/decompress/zstd_ddict.h create mode 100644 lib/zstd/decompress/zstd_decompress.c create mode 100644 lib/zstd/decompress/zstd_decompress_block.c create mode 100644 lib/zstd/decompress/zstd_decompress_block.h create mode 100644 lib/zstd/decompress/zstd_decompress_internal.h create mode 100644 lib/zstd/decompress_sources.h delete mode 100644 lib/zstd/entropy_common.c delete mode 100644 lib/zstd/error_private.h delete mode 100644 lib/zstd/fse.h delete mode 100644 lib/zstd/fse_compress.c delete mode 100644 lib/zstd/fse_decompress.c delete mode 100644 lib/zstd/huf.h delete mode 100644 lib/zstd/huf_compress.c delete mode 100644 lib/zstd/huf_decompress.c delete mode 100644 lib/zstd/mem.h delete mode 100644 lib/zstd/zstd_common.c create mode 100644 lib/zstd/zstd_compress_module.c create mode 100644 lib/zstd/zstd_decompress_module.c delete mode 100644 lib/zstd/zstd_internal.h delete mode 100644 lib/zstd/zstd_opt.h -- 2.28.0 _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel