From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.3 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15C37C43461 for ; Wed, 16 Sep 2020 03:43:27 +0000 (UTC) Received: from lists.sourceforge.net (lists.sourceforge.net [216.105.38.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9999821D94; Wed, 16 Sep 2020 03:43:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=sourceforge.net header.i=@sourceforge.net header.b="jfE60qPJ"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=sf.net header.i=@sf.net header.b="drEW3xaT"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="KE/LPv47" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9999821D94 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=linux-f2fs-devel-bounces@lists.sourceforge.net Received: from [127.0.0.1] (helo=sfs-ml-4.v29.lw.sourceforge.com) by sfs-ml-4.v29.lw.sourceforge.com with esmtp (Exim 4.90_1) (envelope-from ) id 1kIOLu-00076X-Ac; Wed, 16 Sep 2020 03:43:26 +0000 Received: from [172.30.20.202] (helo=mx.sourceforge.net) by sfs-ml-4.v29.lw.sourceforge.com with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kIOLq-000768-Ql; Wed, 16 Sep 2020 03:43:22 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sourceforge.net; s=x; h=Content-Transfer-Encoding:MIME-Version:References: In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=4G5dTWCVv1NOHS3OxfHw4VPNamJGvm8WlcoiqGKDEiA=; b=jfE60qPJQjL+num2wea2zEXq6v 2c2ZPo7ShqnS0Gf42SuUJtNP7c3RKKGLIxBJBbA6no/8ZYBL5r5m2UuXIaY5Ei7UzgxTEMOe6e/32 yeEsiJj5NR0kw/AxmFIVENOKuVKH0gKSi3XPLqrPrB9cGwvTYfrWTJfXnGyGx4FVpyr0=; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=sf.net; s=x ; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=4G5dTWCVv1NOHS3OxfHw4VPNamJGvm8WlcoiqGKDEiA=; b=drEW3xaTriFSvvTbASYPxJbpXo V8elgELq/+gYVblGhN1w7hG36hSyusDgiOUFCahoalO5vul6wbBlXpTRqhEKLIF/8R2ioRjjAG3Tj KpAhoBucHjCNpizT5BbhgwDNpHx9qLuX2AVAsue8xUSVQm5KfSnV/8jCOM+BmLu4WC/o=; Received: from mail-pl1-f196.google.com ([209.85.214.196]) by sfi-mx-3.v28.lw.sourceforge.com with esmtps (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.92.2) id 1kIOLo-00Alce-Bq; Wed, 16 Sep 2020 03:43:22 +0000 Received: by mail-pl1-f196.google.com with SMTP id y6so2440565plt.9; Tue, 15 Sep 2020 20:43:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4G5dTWCVv1NOHS3OxfHw4VPNamJGvm8WlcoiqGKDEiA=; b=KE/LPv47oXVIVzAPme4qL2M2JUEKAM2ruFw4CifP+OrXovLTE3+9OJAdmS1aZffxPs 354JRwOdZ8R7NZY3hm16TAnGV3u68aenDumQ+vfUgmw+/ffLntLbL+A1vrHIO65uUKDo yY/kx2DjivoBs6yeEehtTDpTAqMs3cGz/gtkBy+4wORrONq0GZXesiN1s2UhAltESTJj 2KqpIIcDHPbiC2pzjXsRPbHL8eCdOUTBnfNx/N8f+oHRm25gR21nWp4KeGrtZ41JqiwS MvG3E8nu58j4wEOcSXXFb3ozVyQkC3cEAqCBIhaV+1lOCCZjrAL1Y60weA4iG46KamoQ QS4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4G5dTWCVv1NOHS3OxfHw4VPNamJGvm8WlcoiqGKDEiA=; b=SNMTs1jBOWnQjZZRYTcGMEpaPMrI0s7F+dXhVnNEUX5UGiNxp0HjcdeXcrpDVwUlGG OPqwczdla9bq6CBWSUJdiwuarXzJACK32JSmx2oZECHkPj1HDzdtg8PDloehATQJ07BD Zu0UdfgFVn77/3nRTrdCJ1uH+prmaHfkFR94BuVqTknI7MISC0xnspnHx6SHF7iVMKv0 /h16l4lEYuuzikY5GePdqtIGT5wauS4Gd/0iGnomQaoGkHklXyt9mhgDpB/ogGcnzyVz ATueEgI3JDQGdU5QsgDLt8OO5l+9cQkmh5KMQJ6HCmD67H83GA09xptW1vP/NPMLR8Ph QDRg== X-Gm-Message-State: AOAM533gdtJQUGuIr1lo+OwkoylM0McgpMXDlf7WnmUW+OOMkve+sPq9 Wtpe9lEjySl+WWMYeCC9C0Y= X-Google-Smtp-Source: ABdhPJxxMG1sHnH1sAyJe0cuaB6biIqT0wcpqtHMJ+CXOUKYNB2jPzFXuh7kU4yid2AyFqFG+yCFeQ== X-Received: by 2002:a17:902:8306:b029:d0:cbe1:e7aa with SMTP id bd6-20020a1709028306b02900d0cbe1e7aamr22151055plb.27.1600227787535; Tue, 15 Sep 2020 20:43:07 -0700 (PDT) Received: from nickserv.localdomain (c-98-33-101-203.hsd1.ca.comcast.net. [98.33.101.203]) by smtp.gmail.com with ESMTPSA id i20sm12856635pgk.77.2020.09.15.20.43.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Sep 2020 20:43:07 -0700 (PDT) From: Nick Terrell To: Herbert Xu Date: Tue, 15 Sep 2020 20:43:01 -0700 Message-Id: <20200916034307.2092020-9-nickrterrell@gmail.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200916034307.2092020-1-nickrterrell@gmail.com> References: <20200916034307.2092020-1-nickrterrell@gmail.com> MIME-Version: 1.0 X-Headers-End: 1kIOLo-00Alce-Bq Subject: [f2fs-dev] [PATCH 6/9] f2fs: zstd: Switch to the zstd-1.4.6 API X-BeenThere: linux-f2fs-devel@lists.sourceforge.net X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: squashfs-devel@lists.sourceforge.net, Johannes Weiner , Nick Terrell , Yann Collet , linux-kernel@vger.kernel.org, linux-f2fs-devel@lists.sourceforge.net, Petr Malat , Chris Mason , Nick Terrell , linux-crypto@vger.kernel.org, Kernel Team , Niket Agarwal , linux-btrfs@vger.kernel.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: linux-f2fs-devel-bounces@lists.sourceforge.net From: Nick Terrell Move away from the compatibility wrapper to the zstd-1.4.6 API. This code is more efficient because it uses the single-pass API instead of the streaming API. The streaming API is not necessary because the whole input and output buffers are available. This saves memory because we don't need to allocate a buffer for the window. It is also more efficient because it saves unnecessary memcpy calls. I've had problems testing this code because I see data truncation before and after this patchset. Help testing this patch would be much appreciated. Signed-off-by: Nick Terrell --- fs/f2fs/compress.c | 102 +++++++++++++++++---------------------------- 1 file changed, 38 insertions(+), 64 deletions(-) diff --git a/fs/f2fs/compress.c b/fs/f2fs/compress.c index e056f3a2b404..b79efce81651 100644 --- a/fs/f2fs/compress.c +++ b/fs/f2fs/compress.c @@ -11,7 +11,8 @@ #include #include #include -#include +#include +#include #include "f2fs.h" #include "node.h" @@ -298,21 +299,21 @@ static const struct f2fs_compress_ops f2fs_lz4_ops = { static int zstd_init_compress_ctx(struct compress_ctx *cc) { ZSTD_parameters params; - ZSTD_CStream *stream; + ZSTD_CCtx *ctx; void *workspace; unsigned int workspace_size; params = ZSTD_getParams(F2FS_ZSTD_DEFAULT_CLEVEL, cc->rlen, 0); - workspace_size = ZSTD_CStreamWorkspaceBound(params.cParams); + workspace_size = ZSTD_estimateCCtxSize_usingCParams(params.cParams); workspace = f2fs_kvmalloc(F2FS_I_SB(cc->inode), workspace_size, GFP_NOFS); if (!workspace) return -ENOMEM; - stream = ZSTD_initCStream(params, 0, workspace, workspace_size); - if (!stream) { - printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_initCStream failed\n", + ctx = ZSTD_initStaticCCtx(workspace, workspace_size); + if (!ctx) { + printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_inittaticCStream failed\n", KERN_ERR, F2FS_I_SB(cc->inode)->sb->s_id, __func__); kvfree(workspace); @@ -320,7 +321,7 @@ static int zstd_init_compress_ctx(struct compress_ctx *cc) } cc->private = workspace; - cc->private2 = stream; + cc->private2 = ctx; cc->clen = cc->rlen - PAGE_SIZE - COMPRESS_HEADER_SIZE; return 0; @@ -335,65 +336,48 @@ static void zstd_destroy_compress_ctx(struct compress_ctx *cc) static int zstd_compress_pages(struct compress_ctx *cc) { - ZSTD_CStream *stream = cc->private2; - ZSTD_inBuffer inbuf; - ZSTD_outBuffer outbuf; - int src_size = cc->rlen; - int dst_size = src_size - PAGE_SIZE - COMPRESS_HEADER_SIZE; - int ret; - - inbuf.pos = 0; - inbuf.src = cc->rbuf; - inbuf.size = src_size; - - outbuf.pos = 0; - outbuf.dst = cc->cbuf->cdata; - outbuf.size = dst_size; - - ret = ZSTD_compressStream(stream, &outbuf, &inbuf); - if (ZSTD_isError(ret)) { - printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_compressStream failed, ret: %d\n", - KERN_ERR, F2FS_I_SB(cc->inode)->sb->s_id, - __func__, ZSTD_getErrorCode(ret)); - return -EIO; - } - - ret = ZSTD_endStream(stream, &outbuf); + ZSTD_CCtx *ctx = cc->private2; + const size_t src_size = cc->rlen; + const size_t dst_size = src_size - PAGE_SIZE - COMPRESS_HEADER_SIZE; + ZSTD_parameters params = ZSTD_getParams(F2FS_ZSTD_DEFAULT_CLEVEL, src_size, 0); + size_t ret; + + ret = ZSTD_compress_advanced( + ctx, cc->cbuf->cdata, dst_size, cc->rbuf, src_size, NULL, 0, params); if (ZSTD_isError(ret)) { - printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_endStream returned %d\n", + /* + * there is compressed data remained in intermediate buffer due to + * no more space in cbuf.cdata + */ + if (ZSTD_getErrorCode(ret) == ZSTD_error_dstSize_tooSmall) + return -EAGAIN; + /* other compression errors return -EIO */ + printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_compress_advanced failed, err: %s\n", KERN_ERR, F2FS_I_SB(cc->inode)->sb->s_id, - __func__, ZSTD_getErrorCode(ret)); + __func__, ZSTD_getErrorName(ret)); return -EIO; } - /* - * there is compressed data remained in intermediate buffer due to - * no more space in cbuf.cdata - */ - if (ret) - return -EAGAIN; - - cc->clen = outbuf.pos; + cc->clen = ret; return 0; } static int zstd_init_decompress_ctx(struct decompress_io_ctx *dic) { - ZSTD_DStream *stream; + ZSTD_DCtx *ctx; void *workspace; unsigned int workspace_size; - workspace_size = ZSTD_DStreamWorkspaceBound(MAX_COMPRESS_WINDOW_SIZE); + workspace_size = ZSTD_estimateDCtxSize(); workspace = f2fs_kvmalloc(F2FS_I_SB(dic->inode), workspace_size, GFP_NOFS); if (!workspace) return -ENOMEM; - stream = ZSTD_initDStream(MAX_COMPRESS_WINDOW_SIZE, - workspace, workspace_size); - if (!stream) { - printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_initDStream failed\n", + ctx = ZSTD_initStaticDCtx(workspace, workspace_size); + if (!ctx) { + printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_initStaticDCtx failed\n", KERN_ERR, F2FS_I_SB(dic->inode)->sb->s_id, __func__); kvfree(workspace); @@ -401,7 +385,7 @@ static int zstd_init_decompress_ctx(struct decompress_io_ctx *dic) } dic->private = workspace; - dic->private2 = stream; + dic->private2 = ctx; return 0; } @@ -415,28 +399,18 @@ static void zstd_destroy_decompress_ctx(struct decompress_io_ctx *dic) static int zstd_decompress_pages(struct decompress_io_ctx *dic) { - ZSTD_DStream *stream = dic->private2; - ZSTD_inBuffer inbuf; - ZSTD_outBuffer outbuf; - int ret; - - inbuf.pos = 0; - inbuf.src = dic->cbuf->cdata; - inbuf.size = dic->clen; - - outbuf.pos = 0; - outbuf.dst = dic->rbuf; - outbuf.size = dic->rlen; + ZSTD_DCtx *ctx = dic->private2; + size_t ret; - ret = ZSTD_decompressStream(stream, &outbuf, &inbuf); + ret = ZSTD_decompressDCtx(ctx, dic->rbuf, dic->rlen, dic->cbuf->cdata, dic->clen); if (ZSTD_isError(ret)) { - printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_compressStream failed, ret: %d\n", + printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_decompressDCtx failed, err: %s\n", KERN_ERR, F2FS_I_SB(dic->inode)->sb->s_id, - __func__, ZSTD_getErrorCode(ret)); + __func__, ZSTD_getErrorName(ret)); return -EIO; } - if (dic->rlen != outbuf.pos) { + if (dic->rlen != ret) { printk_ratelimited("%sF2FS-fs (%s): %s ZSTD invalid rlen:%zu, " "expected:%lu\n", KERN_ERR, F2FS_I_SB(dic->inode)->sb->s_id, -- 2.28.0 _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel