From: Nick Terrell <nickrterrell@gmail.com>
To: Herbert Xu <herbert@gondor.apana.org.au>
Cc: linux-crypto@vger.kernel.org, linux-btrfs@vger.kernel.org,
squashfs-devel@lists.sourceforge.net,
linux-f2fs-devel@lists.sourceforge.net,
linux-kernel@vger.kernel.org, Kernel Team <Kernel-team@fb.com>,
Nick Terrell <nickrterrell@gmail.com>,
Nick Terrell <terrelln@fb.com>, Chris Mason <clm@fb.com>,
Petr Malat <oss@malat.biz>, Johannes Weiner <jweiner@fb.com>,
Niket Agarwal <niketa@fb.com>, Yann Collet <cyan@fb.com>
Subject: [PATCH 8/9] lib: unzstd: Switch to the zstd-1.4.6 API
Date: Tue, 15 Sep 2020 20:43:05 -0700 [thread overview]
Message-ID: <20200916034307.2092020-13-nickrterrell@gmail.com> (raw)
In-Reply-To: <20200916034307.2092020-1-nickrterrell@gmail.com>
From: Nick Terrell <terrelln@fb.com>
Move away from the compatibility wrapper to the zstd-1.4.6 API. This
code is functionally equivalent.
Signed-off-by: Nick Terrell <terrelln@fb.com>
---
lib/decompress_unzstd.c | 40 ++++++++++++++--------------------------
1 file changed, 14 insertions(+), 26 deletions(-)
diff --git a/lib/decompress_unzstd.c b/lib/decompress_unzstd.c
index a79f705f236d..d4685df0e120 100644
--- a/lib/decompress_unzstd.c
+++ b/lib/decompress_unzstd.c
@@ -73,7 +73,8 @@
#include <linux/decompress/mm.h>
#include <linux/kernel.h>
-#include <linux/zstd_compat.h>
+#include <linux/zstd.h>
+#include <linux/zstd_errors.h>
/* 128MB is the maximum window size supported by zstd. */
#define ZSTD_WINDOWSIZE_MAX (1 << ZSTD_WINDOWLOG_MAX)
@@ -120,9 +121,9 @@ static int INIT decompress_single(const u8 *in_buf, long in_len, u8 *out_buf,
long out_len, long *in_pos,
void (*error)(char *x))
{
- const size_t wksp_size = ZSTD_DCtxWorkspaceBound();
+ const size_t wksp_size = ZSTD_estimateDCtxSize();
void *wksp = large_malloc(wksp_size);
- ZSTD_DCtx *dctx = ZSTD_initDCtx(wksp, wksp_size);
+ ZSTD_DCtx *dctx = ZSTD_initStaticDCtx(wksp, wksp_size);
int err;
size_t ret;
@@ -165,7 +166,6 @@ static int INIT __unzstd(unsigned char *in_buf, long in_len,
{
ZSTD_inBuffer in;
ZSTD_outBuffer out;
- ZSTD_frameParams params;
void *in_allocated = NULL;
void *out_allocated = NULL;
void *wksp = NULL;
@@ -229,36 +229,24 @@ static int INIT __unzstd(unsigned char *in_buf, long in_len,
out.size = out_len;
/*
- * We need to know the window size to allocate the ZSTD_DStream.
- * Since we are streaming, we need to allocate a buffer for the sliding
- * window. The window size varies from 1 KB to ZSTD_WINDOWSIZE_MAX
- * (8 MB), so it is important to use the actual value so as not to
- * waste memory when it is smaller.
+ * Zstd determines the workspace size from the window size written
+ * into the frame header. This ensures that we use the minimum value
+ * possible, since the window size varies from 1 KB to ZSTD_WINDOWSIZE_MAX
+ * (1 GB), so it is very important to use the actual value.
*/
- ret = ZSTD_getFrameParams(¶ms, in.src, in.size);
+ wksp_size = ZSTD_estimateDStreamSize_fromFrame(in.src, in.size);
err = handle_zstd_error(ret, error);
if (err)
goto out;
- if (ret != 0) {
- error("ZSTD-compressed data has an incomplete frame header");
- err = -1;
- goto out;
- }
- if (params.windowSize > ZSTD_WINDOWSIZE_MAX) {
- error("ZSTD-compressed data has too large a window size");
+ wksp = large_malloc(wksp_size);
+ if (wksp == NULL) {
+ error("Out of memory while allocating ZSTD_DStream");
err = -1;
goto out;
}
-
- /*
- * Allocate the ZSTD_DStream now that we know how much memory is
- * required.
- */
- wksp_size = ZSTD_DStreamWorkspaceBound(params.windowSize);
- wksp = large_malloc(wksp_size);
- dstream = ZSTD_initDStream(params.windowSize, wksp, wksp_size);
+ dstream = ZSTD_initStaticDStream(wksp, wksp_size);
if (dstream == NULL) {
- error("Out of memory while allocating ZSTD_DStream");
+ error("ZSTD_initStaticDStream failed");
err = -1;
goto out;
}
--
2.28.0
WARNING: multiple messages have this Message-ID (diff)
From: Nick Terrell <nickrterrell@gmail.com>
To: Herbert Xu <herbert@gondor.apana.org.au>
Cc: squashfs-devel@lists.sourceforge.net,
Johannes Weiner <jweiner@fb.com>,
Nick Terrell <nickrterrell@gmail.com>, Yann Collet <cyan@fb.com>,
linux-kernel@vger.kernel.org,
linux-f2fs-devel@lists.sourceforge.net,
Petr Malat <oss@malat.biz>, Chris Mason <clm@fb.com>,
Nick Terrell <terrelln@fb.com>,
linux-crypto@vger.kernel.org, Kernel Team <Kernel-team@fb.com>,
Niket Agarwal <niketa@fb.com>,
linux-btrfs@vger.kernel.org
Subject: [f2fs-dev] [PATCH 8/9] lib: unzstd: Switch to the zstd-1.4.6 API
Date: Tue, 15 Sep 2020 20:43:05 -0700 [thread overview]
Message-ID: <20200916034307.2092020-13-nickrterrell@gmail.com> (raw)
In-Reply-To: <20200916034307.2092020-1-nickrterrell@gmail.com>
From: Nick Terrell <terrelln@fb.com>
Move away from the compatibility wrapper to the zstd-1.4.6 API. This
code is functionally equivalent.
Signed-off-by: Nick Terrell <terrelln@fb.com>
---
lib/decompress_unzstd.c | 40 ++++++++++++++--------------------------
1 file changed, 14 insertions(+), 26 deletions(-)
diff --git a/lib/decompress_unzstd.c b/lib/decompress_unzstd.c
index a79f705f236d..d4685df0e120 100644
--- a/lib/decompress_unzstd.c
+++ b/lib/decompress_unzstd.c
@@ -73,7 +73,8 @@
#include <linux/decompress/mm.h>
#include <linux/kernel.h>
-#include <linux/zstd_compat.h>
+#include <linux/zstd.h>
+#include <linux/zstd_errors.h>
/* 128MB is the maximum window size supported by zstd. */
#define ZSTD_WINDOWSIZE_MAX (1 << ZSTD_WINDOWLOG_MAX)
@@ -120,9 +121,9 @@ static int INIT decompress_single(const u8 *in_buf, long in_len, u8 *out_buf,
long out_len, long *in_pos,
void (*error)(char *x))
{
- const size_t wksp_size = ZSTD_DCtxWorkspaceBound();
+ const size_t wksp_size = ZSTD_estimateDCtxSize();
void *wksp = large_malloc(wksp_size);
- ZSTD_DCtx *dctx = ZSTD_initDCtx(wksp, wksp_size);
+ ZSTD_DCtx *dctx = ZSTD_initStaticDCtx(wksp, wksp_size);
int err;
size_t ret;
@@ -165,7 +166,6 @@ static int INIT __unzstd(unsigned char *in_buf, long in_len,
{
ZSTD_inBuffer in;
ZSTD_outBuffer out;
- ZSTD_frameParams params;
void *in_allocated = NULL;
void *out_allocated = NULL;
void *wksp = NULL;
@@ -229,36 +229,24 @@ static int INIT __unzstd(unsigned char *in_buf, long in_len,
out.size = out_len;
/*
- * We need to know the window size to allocate the ZSTD_DStream.
- * Since we are streaming, we need to allocate a buffer for the sliding
- * window. The window size varies from 1 KB to ZSTD_WINDOWSIZE_MAX
- * (8 MB), so it is important to use the actual value so as not to
- * waste memory when it is smaller.
+ * Zstd determines the workspace size from the window size written
+ * into the frame header. This ensures that we use the minimum value
+ * possible, since the window size varies from 1 KB to ZSTD_WINDOWSIZE_MAX
+ * (1 GB), so it is very important to use the actual value.
*/
- ret = ZSTD_getFrameParams(¶ms, in.src, in.size);
+ wksp_size = ZSTD_estimateDStreamSize_fromFrame(in.src, in.size);
err = handle_zstd_error(ret, error);
if (err)
goto out;
- if (ret != 0) {
- error("ZSTD-compressed data has an incomplete frame header");
- err = -1;
- goto out;
- }
- if (params.windowSize > ZSTD_WINDOWSIZE_MAX) {
- error("ZSTD-compressed data has too large a window size");
+ wksp = large_malloc(wksp_size);
+ if (wksp == NULL) {
+ error("Out of memory while allocating ZSTD_DStream");
err = -1;
goto out;
}
-
- /*
- * Allocate the ZSTD_DStream now that we know how much memory is
- * required.
- */
- wksp_size = ZSTD_DStreamWorkspaceBound(params.windowSize);
- wksp = large_malloc(wksp_size);
- dstream = ZSTD_initDStream(params.windowSize, wksp, wksp_size);
+ dstream = ZSTD_initStaticDStream(wksp, wksp_size);
if (dstream == NULL) {
- error("Out of memory while allocating ZSTD_DStream");
+ error("ZSTD_initStaticDStream failed");
err = -1;
goto out;
}
--
2.28.0
_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
next prev parent reply other threads:[~2020-09-16 3:44 UTC|newest]
Thread overview: 74+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-09-16 3:42 [PATCH 0/9] Update to zstd-1.4.6 Nick Terrell
2020-09-16 3:42 ` [f2fs-dev] " Nick Terrell
2020-09-16 3:42 ` [PATCH 1/9] lib: zstd: Add zstd compatibility wrapper Nick Terrell
2020-09-16 3:42 ` [f2fs-dev] " Nick Terrell
2020-09-16 8:48 ` Christoph Hellwig
2020-09-16 8:48 ` [f2fs-dev] " Christoph Hellwig
2020-09-16 20:21 ` Nick Terrell
2020-09-16 20:21 ` [f2fs-dev] " Nick Terrell via Linux-f2fs-devel
2020-09-16 3:42 ` [PATCH 2/9] lib: zstd: Add decompress_sources.h for decompress_unzstd Nick Terrell
2020-09-16 3:42 ` [f2fs-dev] " Nick Terrell
2020-09-16 3:42 ` [f2fs-dev] [PATCH 3/9] lib: zstd: Upgrade to latest upstream zstd version 1.4.6 Nick Terrell
2020-09-16 3:42 ` Nick Terrell
2020-09-16 8:01 ` kernel test robot
2020-09-16 8:01 ` kernel test robot
2020-09-16 8:01 ` kernel test robot
2020-09-16 20:53 ` Nick Terrell
2020-09-16 20:53 ` [f2fs-dev] " Nick Terrell via Linux-f2fs-devel
2020-09-16 3:42 ` [PATCH 4/9] crypto: zstd: Switch to zstd-1.4.6 API Nick Terrell
2020-09-16 3:42 ` [f2fs-dev] " Nick Terrell
2020-09-16 8:49 ` Christoph Hellwig
2020-09-16 8:49 ` [f2fs-dev] " Christoph Hellwig
2020-09-16 3:42 ` [PATCH 5/9] btrfs: zstd: Switch to the " Nick Terrell
2020-09-16 3:42 ` [f2fs-dev] " Nick Terrell
2020-09-16 8:49 ` Christoph Hellwig
2020-09-16 8:49 ` [f2fs-dev] " Christoph Hellwig
2020-09-16 14:20 ` Chris Mason
2020-09-16 14:20 ` [f2fs-dev] " Chris Mason via Linux-f2fs-devel
2020-09-16 14:30 ` Christoph Hellwig
2020-09-16 14:30 ` [f2fs-dev] " Christoph Hellwig
2020-09-16 14:43 ` Chris Mason
2020-09-16 14:43 ` [f2fs-dev] " Chris Mason via Linux-f2fs-devel
2020-09-16 14:46 ` Christoph Hellwig
2020-09-16 14:46 ` [f2fs-dev] " Christoph Hellwig
2020-09-16 15:01 ` Chris Mason
2020-09-16 15:01 ` [f2fs-dev] " Chris Mason via Linux-f2fs-devel
2020-09-16 18:27 ` Eric Biggers
2020-09-16 18:27 ` [f2fs-dev] " Eric Biggers
2020-09-16 19:18 ` Nick Terrell
2020-09-16 19:18 ` [f2fs-dev] " Nick Terrell via Linux-f2fs-devel
2020-09-17 1:35 ` Rik van Riel
2020-09-17 10:04 ` Christoph Hellwig
2020-09-17 10:04 ` [f2fs-dev] " Christoph Hellwig
2020-09-17 14:28 ` Chris Mason
2020-09-17 14:28 ` [f2fs-dev] " Chris Mason via Linux-f2fs-devel
2020-09-17 17:57 ` Nick Terrell
2020-09-17 17:57 ` [f2fs-dev] " Nick Terrell via Linux-f2fs-devel
2020-09-16 3:43 ` [PATCH 6/9] f2fs: " Nick Terrell
2020-09-16 3:43 ` [f2fs-dev] " Nick Terrell
2020-09-16 6:31 ` Chao Yu
2020-09-16 6:31 ` Chao Yu
2020-09-16 18:39 ` [f2fs-dev] " Nick Terrell via Linux-f2fs-devel
2020-09-16 18:39 ` Nick Terrell
2020-09-17 6:31 ` [f2fs-dev] " Chao Yu
2020-09-17 6:31 ` Chao Yu
2020-09-17 18:00 ` [f2fs-dev] " Nick Terrell via Linux-f2fs-devel
2020-09-17 18:00 ` Nick Terrell
2020-09-17 19:34 ` [f2fs-dev] " Nick Terrell via Linux-f2fs-devel
2020-09-17 19:34 ` Nick Terrell
2020-09-18 1:47 ` [f2fs-dev] " Chao Yu
2020-09-18 1:47 ` Chao Yu
2020-09-18 2:56 ` [f2fs-dev] " Eric Biggers
2020-09-18 2:56 ` Eric Biggers
2020-09-18 2:58 ` [f2fs-dev] " Chao Yu
2020-09-18 2:58 ` Chao Yu
2020-09-18 5:39 ` [f2fs-dev] " Nick Terrell via Linux-f2fs-devel
2020-09-18 5:39 ` Nick Terrell
2020-09-18 1:41 ` [f2fs-dev] " Chao Yu
2020-09-18 1:41 ` Chao Yu
2020-09-16 3:43 ` [PATCH 7/9] squashfs: " Nick Terrell
2020-09-16 3:43 ` [f2fs-dev] " Nick Terrell
2020-09-16 3:43 ` Nick Terrell [this message]
2020-09-16 3:43 ` [f2fs-dev] [PATCH 8/9] lib: unzstd: " Nick Terrell
2020-09-16 3:43 ` [PATCH 9/9] lib: zstd: Remove zstd compatibility wrapper Nick Terrell
2020-09-16 3:43 ` [f2fs-dev] " Nick Terrell
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200916034307.2092020-13-nickrterrell@gmail.com \
--to=nickrterrell@gmail.com \
--cc=Kernel-team@fb.com \
--cc=clm@fb.com \
--cc=cyan@fb.com \
--cc=herbert@gondor.apana.org.au \
--cc=jweiner@fb.com \
--cc=linux-btrfs@vger.kernel.org \
--cc=linux-crypto@vger.kernel.org \
--cc=linux-f2fs-devel@lists.sourceforge.net \
--cc=linux-kernel@vger.kernel.org \
--cc=niketa@fb.com \
--cc=oss@malat.biz \
--cc=squashfs-devel@lists.sourceforge.net \
--cc=terrelln@fb.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.