From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.3 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7580AC388F2 for ; Tue, 3 Nov 2020 06:02:49 +0000 (UTC) Received: from lists.sourceforge.net (lists.sourceforge.net [216.105.38.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0853D22277; Tue, 3 Nov 2020 06:02:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=sourceforge.net header.i=@sourceforge.net header.b="ZGKbfSKT"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=sf.net header.i=@sf.net header.b="EjO7rRW5"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="gSSV2Azc" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0853D22277 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=linux-f2fs-devel-bounces@lists.sourceforge.net Received: from [127.0.0.1] (helo=sfs-ml-1.v29.lw.sourceforge.com) by sfs-ml-1.v29.lw.sourceforge.com with esmtp (Exim 4.90_1) (envelope-from ) id 1kZpP5-00084D-JD; Tue, 03 Nov 2020 06:02:47 +0000 Received: from [172.30.20.202] (helo=mx.sourceforge.net) by sfs-ml-1.v29.lw.sourceforge.com with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kZpOw-00082V-9J; Tue, 03 Nov 2020 06:02:38 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sourceforge.net; s=x; h=Content-Transfer-Encoding:MIME-Version:References: In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=tfz5JoPFBe+5NZSwhHeml9GZoHiera3KaxD144JrSCQ=; b=ZGKbfSKTO9Z3ia76Br8LhEJJcX 5d7hM0FGLG5mO5l/i5O6Ie+7OxWFTMKc8+AbeTvIn8rsMyQh2jxjBvPdgqQ7Eq8Toc+f2lyhSJhPY ijhua9ri6CLj+aG4XtSBc2dk5vaO4E/kSC4S+ivH10gEttLot0vJQiY8FCwcuRHBJVv0=; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sf.net; s=x ; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=tfz5JoPFBe+5NZSwhHeml9GZoHiera3KaxD144JrSCQ=; b=EjO7rRW5T9pSnnB45t2cdhEg8w OexV5ucctRT2H9bqmyysjAX/q2fM5WKYiumGBTc3LT4Qkvsh2sPGU3Jinr1K1MINM5hyeV1Mg7tM8 vDL33gDhs0MBNiAwJWaYMHxjbl+8d4vWmopcLktT0GpnMPgTfhmgOXtdFvAE8F8q/kVU=; Received: from mail-pf1-f194.google.com ([209.85.210.194]) by sfi-mx-1.v28.lw.sourceforge.com with esmtps (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.92.2) id 1kZpOr-007KjA-Cd; Tue, 03 Nov 2020 06:02:38 +0000 Received: by mail-pf1-f194.google.com with SMTP id x13so13247004pfa.9; Mon, 02 Nov 2020 22:02:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tfz5JoPFBe+5NZSwhHeml9GZoHiera3KaxD144JrSCQ=; b=gSSV2AzcSgKjWbA0/duEx/32yEqKcFXKR17Ctdoe1pcX/lfA9NvUrSBAIAJSl1Z7tj H2ui0Evi/5PTqJlwsDtcpVN419Jbs3OELOwS5RGAVngk2jRvNTKb/ZkAsm8Oe0/fXi4+ ckqV6Ge2q4B+j+A0/CRILjnvPQQHcR9Nw2Igf4qVYN/H3Rf8+LYVEcSdySSXengisu8U 64/db4Qsod9qGVW7k1Py7tK0/mg2MC2id1BrP7MklQRcBwyF3leReGVmsyiWK2RL8ZT/ d3x+aGj5IWmdxpL1LHBxjSXE4bFbeCYbSpdJXO5s+2vkwRlfEyRmCqBIVGgf8SoSrYxP NLxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tfz5JoPFBe+5NZSwhHeml9GZoHiera3KaxD144JrSCQ=; b=YN0Y83FQ4Ft2Tv2Qsf/mxjKuvenL0IE99iuCcgVc0OEtu0XfMEK4aYqnH3cUyVPRGf lp/vU6OgidszYOX79eLyIeSQrNIJAY3a1leOeMm+j8JrLTThi2gPyUNV1LjWAbW5fFSH SxJcVUKfnT0fLEPyzqB1e5hO3Y39jFBxvMJ2Al8PvLhBdXg04ONOshcSZ697rB5H68Gw pghfhg5Fw6nxJe1rqnt5DERX71AZZD31wbqAufR4gVGKCbXvF+VpZz61NHE2hVU8YOL+ S7qWKesKWBdD9ImKLxslTR5Fm5e+/5jyk+PKdxk9WE8GHE5SrXKK0yvMfseIMWLLltH0 Bqog== X-Gm-Message-State: AOAM530VWlNSa6u3S12pPgKVt7kxaY/TD7umhxL4eaXnfX1ERWM1uEv4 O7w0ySjXqhcVD+Tg9+Vr7Vs= X-Google-Smtp-Source: ABdhPJypwJ893RanOWhi2oqXdSk3EUdYWLWR9gKh4oGpMiSgrcRpGfTkc49X7X3lz+5ApmK445SP+Q== X-Received: by 2002:a63:389:: with SMTP id 131mr3966874pgd.128.1604383340660; Mon, 02 Nov 2020 22:02:20 -0800 (PST) Received: from nickserv.localdomain (c-98-33-101-203.hsd1.ca.comcast.net. [98.33.101.203]) by smtp.gmail.com with ESMTPSA id b16sm15647269pfp.195.2020.11.02.22.02.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Nov 2020 22:02:20 -0800 (PST) From: Nick Terrell To: Herbert Xu Date: Mon, 2 Nov 2020 22:05:32 -0800 Message-Id: <20201103060535.8460-7-nickrterrell@gmail.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201103060535.8460-1-nickrterrell@gmail.com> References: <20201103060535.8460-1-nickrterrell@gmail.com> MIME-Version: 1.0 X-Headers-End: 1kZpOr-007KjA-Cd Subject: [f2fs-dev] [PATCH v5 6/9] f2fs: zstd: Switch to the zstd-1.4.6 API X-BeenThere: linux-f2fs-devel@lists.sourceforge.net X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: squashfs-devel@lists.sourceforge.net, Johannes Weiner , Nick Terrell , Yann Collet , linux-kernel@vger.kernel.org, linux-f2fs-devel@lists.sourceforge.net, Petr Malat , Chris Mason , Nick Terrell , linux-crypto@vger.kernel.org, Kernel Team , Niket Agarwal , linux-btrfs@vger.kernel.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: linux-f2fs-devel-bounces@lists.sourceforge.net From: Nick Terrell Move away from the compatibility wrapper to the zstd-1.4.6 API. This code is more efficient because it uses the single-pass API instead of the streaming API. The streaming API is not necessary because the whole input and output buffers are available. This saves memory because we don't need to allocate a buffer for the window. It is also more efficient because it saves unnecessary memcpy calls. Compression memory increases from 168 KB to 204 KB because upstream uses slightly more memory. Decompression memory decreases from 1.4 MB to 158 KB. Signed-off-by: Nick Terrell --- fs/f2fs/compress.c | 101 +++++++++++++++++---------------------------- 1 file changed, 37 insertions(+), 64 deletions(-) diff --git a/fs/f2fs/compress.c b/fs/f2fs/compress.c index 57a6360b9827..8f8234877666 100644 --- a/fs/f2fs/compress.c +++ b/fs/f2fs/compress.c @@ -11,7 +11,8 @@ #include #include #include -#include +#include +#include #include "f2fs.h" #include "node.h" @@ -322,21 +323,21 @@ static const struct f2fs_compress_ops f2fs_lz4_ops = { static int zstd_init_compress_ctx(struct compress_ctx *cc) { ZSTD_parameters params; - ZSTD_CStream *stream; + ZSTD_CCtx *ctx; void *workspace; unsigned int workspace_size; params = ZSTD_getParams(F2FS_ZSTD_DEFAULT_CLEVEL, cc->rlen, 0); - workspace_size = ZSTD_CStreamWorkspaceBound(params.cParams); + workspace_size = ZSTD_estimateCCtxSize_usingCParams(params.cParams); workspace = f2fs_kvmalloc(F2FS_I_SB(cc->inode), workspace_size, GFP_NOFS); if (!workspace) return -ENOMEM; - stream = ZSTD_initCStream(params, 0, workspace, workspace_size); - if (!stream) { - printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_initCStream failed\n", + ctx = ZSTD_initStaticCCtx(workspace, workspace_size); + if (!ctx) { + printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_inittaticCStream failed\n", KERN_ERR, F2FS_I_SB(cc->inode)->sb->s_id, __func__); kvfree(workspace); @@ -344,7 +345,7 @@ static int zstd_init_compress_ctx(struct compress_ctx *cc) } cc->private = workspace; - cc->private2 = stream; + cc->private2 = ctx; cc->clen = cc->rlen - PAGE_SIZE - COMPRESS_HEADER_SIZE; return 0; @@ -359,66 +360,48 @@ static void zstd_destroy_compress_ctx(struct compress_ctx *cc) static int zstd_compress_pages(struct compress_ctx *cc) { - ZSTD_CStream *stream = cc->private2; - ZSTD_inBuffer inbuf; - ZSTD_outBuffer outbuf; - int src_size = cc->rlen; - int dst_size = src_size - PAGE_SIZE - COMPRESS_HEADER_SIZE; - int ret; - - inbuf.pos = 0; - inbuf.src = cc->rbuf; - inbuf.size = src_size; - - outbuf.pos = 0; - outbuf.dst = cc->cbuf->cdata; - outbuf.size = dst_size; + ZSTD_CCtx *ctx = cc->private2; + const size_t src_size = cc->rlen; + const size_t dst_size = src_size - PAGE_SIZE - COMPRESS_HEADER_SIZE; + ZSTD_parameters params = ZSTD_getParams(F2FS_ZSTD_DEFAULT_CLEVEL, src_size, 0); + size_t ret; - ret = ZSTD_compressStream(stream, &outbuf, &inbuf); + ret = ZSTD_compress_advanced( + ctx, cc->cbuf->cdata, dst_size, cc->rbuf, src_size, NULL, 0, params); if (ZSTD_isError(ret)) { - printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_compressStream failed, ret: %d\n", - KERN_ERR, F2FS_I_SB(cc->inode)->sb->s_id, - __func__, ZSTD_getErrorCode(ret)); - return -EIO; - } - - ret = ZSTD_endStream(stream, &outbuf); - if (ZSTD_isError(ret)) { - printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_endStream returned %d\n", + /* + * there is compressed data remained in intermediate buffer due to + * no more space in cbuf.cdata + */ + if (ZSTD_getErrorCode(ret) == ZSTD_error_dstSize_tooSmall) + return -EAGAIN; + /* other compression errors return -EIO */ + printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_compress_advanced failed, err: %s\n", KERN_ERR, F2FS_I_SB(cc->inode)->sb->s_id, - __func__, ZSTD_getErrorCode(ret)); + __func__, ZSTD_getErrorName(ret)); return -EIO; } - /* - * there is compressed data remained in intermediate buffer due to - * no more space in cbuf.cdata - */ - if (ret) - return -EAGAIN; - - cc->clen = outbuf.pos; + cc->clen = ret; return 0; } static int zstd_init_decompress_ctx(struct decompress_io_ctx *dic) { - ZSTD_DStream *stream; + ZSTD_DCtx *ctx; void *workspace; unsigned int workspace_size; - unsigned int max_window_size = - MAX_COMPRESS_WINDOW_SIZE(dic->log_cluster_size); - workspace_size = ZSTD_DStreamWorkspaceBound(max_window_size); + workspace_size = ZSTD_estimateDCtxSize(); workspace = f2fs_kvmalloc(F2FS_I_SB(dic->inode), workspace_size, GFP_NOFS); if (!workspace) return -ENOMEM; - stream = ZSTD_initDStream(max_window_size, workspace, workspace_size); - if (!stream) { - printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_initDStream failed\n", + ctx = ZSTD_initStaticDCtx(workspace, workspace_size); + if (!ctx) { + printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_initStaticDCtx failed\n", KERN_ERR, F2FS_I_SB(dic->inode)->sb->s_id, __func__); kvfree(workspace); @@ -426,7 +409,7 @@ static int zstd_init_decompress_ctx(struct decompress_io_ctx *dic) } dic->private = workspace; - dic->private2 = stream; + dic->private2 = ctx; return 0; } @@ -440,28 +423,18 @@ static void zstd_destroy_decompress_ctx(struct decompress_io_ctx *dic) static int zstd_decompress_pages(struct decompress_io_ctx *dic) { - ZSTD_DStream *stream = dic->private2; - ZSTD_inBuffer inbuf; - ZSTD_outBuffer outbuf; - int ret; - - inbuf.pos = 0; - inbuf.src = dic->cbuf->cdata; - inbuf.size = dic->clen; - - outbuf.pos = 0; - outbuf.dst = dic->rbuf; - outbuf.size = dic->rlen; + ZSTD_DCtx *ctx = dic->private2; + size_t ret; - ret = ZSTD_decompressStream(stream, &outbuf, &inbuf); + ret = ZSTD_decompressDCtx(ctx, dic->rbuf, dic->rlen, dic->cbuf->cdata, dic->clen); if (ZSTD_isError(ret)) { - printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_compressStream failed, ret: %d\n", + printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_decompressDCtx failed, err: %s\n", KERN_ERR, F2FS_I_SB(dic->inode)->sb->s_id, - __func__, ZSTD_getErrorCode(ret)); + __func__, ZSTD_getErrorName(ret)); return -EIO; } - if (dic->rlen != outbuf.pos) { + if (dic->rlen != ret) { printk_ratelimited("%sF2FS-fs (%s): %s ZSTD invalid rlen:%zu, " "expected:%lu\n", KERN_ERR, F2FS_I_SB(dic->inode)->sb->s_id, -- 2.28.0 _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel