From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42A50C433EF for ; Tue, 26 Jul 2022 05:22:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237512AbiGZFWk (ORCPT ); Tue, 26 Jul 2022 01:22:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34592 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231258AbiGZFWj (ORCPT ); Tue, 26 Jul 2022 01:22:39 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C32C327CD5 for ; Mon, 25 Jul 2022 22:22:38 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 7CD311F9C3; Tue, 26 Jul 2022 05:22:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1658812957; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=N5vMrlZsznWTxDDP5VjzMo40WkvisIyfwBvd8HxY+Lw=; b=UtLL8nr+Rb2hlOoLtPbEKf4D+iLxk3NNRqZdx2ka85l+XKRZQ8926YRnzyfwqTtvgIxrlC CSkeXFdKaNwPFNor+94W2IaCjkNHP8asK/hxEmBYfKYKs/FsAX8qLiR7S5YsPkQ3OTHNQb Tbf93y8ovMV8xT/aka+2l92fJ5VFRLE= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 7B8D013A12; Tue, 26 Jul 2022 05:22:35 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 5Ep5Ext632IFOwAAMHmgww (envelope-from ); Tue, 26 Jul 2022 05:22:35 +0000 From: Qu Wenruo To: u-boot@lists.denx.de Cc: marek.behun@nic.cz, linux-btrfs@vger.kernel.org, jnhuang95@gmail.com, linux-erofs@lists.ozlabs.org, trini@konsulko.com, joaomarcos.costa@bootlin.com, thomas.petazzoni@bootlin.com, miquel.raynal@bootlin.com Subject: [PATCH v2 0/8] U-boot: fs: add generic unaligned read offset handling Date: Tue, 26 Jul 2022 13:22:08 +0800 Message-Id: X-Mailer: git-send-email 2.37.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org [CHANGELOG] v2->v1: - Fix a linkage error where (U64 % U32) is called without proper helper Fix it with U64 & (U32 - 1), as the U32 value (@blocksize) should always be power of 2, thus (@blocksize - 1) is the mask we want to calculate the offset inside the block. Above change only affects the 4th patch, everything else is not touched. RFC->v1: - More (manual) testing Unfortunately, in the latest master (75967970850a), the fs-tests.sh always seems to hang at preparing the fs image. Thus still has to do manual testing, tested btrfs, ext4 and fat, with aligned and unaligned read, also added soft link read, all looks fine here. Extra testing is still appreciated. - Two more btrfs specific bug fixes All exposed during manual tests - Remove the tailing unaligned block handling In fact, all fses can easily handle such case, just a min() call is enough. - Remove the support for sandboxfs Since it's using read() calls, really no need to do block alignment check. - Enhanced blocksize check Ensure the returned blocksize is not only non-error, but also non-zero. This patchset can be fetched from github: https://github.com/adam900710/u-boot/tree/fs_unaligned_read [BACKGROUND] Unlike FUSE/Kernel which always pass aligned read range, U-boot fs code just pass the request range to underlying fses. Under most case, this works fine, as U-boot only really needs to read the whole file (aka, 0 for both offset and len, len will be later determined using file size). But if some advanced user/script wants to extract kernel/initramfs from combined image, we may need to do unaligned read in that case. [ADVANTAGE] This patchset will handle unaligned read range in _fs_read(): - Get blocksize of the underlying fs - Read the leading block contianing the unaligned range The full block will be stored in a local buffer, then only copy the bytes in the unaligned range into the destination buffer. If the first block covers the whole range, we just call it aday. - Read the remaining range which starts at block aligned offset For most common case, which is 0 offset and 0 len, the code will not be changed at all. Just one extra get_blocksize() call, and for FAT/Btrfs/EROFS they all have cached blocksize, thus it takes almost no extra cost. Although for EXT4, it doesn't seem to cache the blocksize globally, thus has to do a path resolve and grab the blocksize. [DISADVANTAGE] The involved problem is: - Extra path resolving All those supported fs may have to do one extra path resolve if the read offset is not aligned. For EXT4, it will do one extra path resolve just to grab the blocksize. For data read which starts at offset 0 (the most common case), it should cause *NO* difference in performance. As the extra handling is only for unaligned offset. The common path is not really modified. [SUPPORTED FSES] - Btrfs (manually tested*) - Ext4 (manually tested) - FAT (manually tested) - Erofs - ubifs (unable to test, due to compile failure) *: Failed to get the test cases run, thus have to go sandbox mode, and attach an image with target fs, load the target file (with unaligned range) and compare the result using md5sum. For EXT4/FAT, they may need extra cleanup, as their existing unaligned range handling is no longer needed anymore, cleaning them up should free more code lines than the added one. Just not confident enough to modify them all by myself. [UNSUPPORTED FSES] - Squashfs They don't support non-zero offset, thus it can not handle the block aligned range. Need extra help to add block aligned offset support. - Semihostfs - Sandboxfs They all use read() directly, no need to do alignment check at all. Extra testing/feedback is always appreciated. Qu Wenruo (8): fs: fat: unexport file_fat_read_at() fs: btrfs: fix a bug which no data get read if the length is not 0 fs: btrfs: fix a crash if specified range is beyond file size fs: btrfs: move the unaligned read code to _fs_read() for btrfs fs: ext4: rely on _fs_read() to handle leading unaligned block read fs: fat: rely on higher layer to get block aligned read range fs: ubifs: rely on higher layer to do unaligned read fs: erofs: add unaligned read range handling fs/btrfs/btrfs.c | 33 ++++++++--- fs/btrfs/inode.c | 89 +++-------------------------- fs/erofs/internal.h | 1 + fs/erofs/super.c | 6 ++ fs/ext4/ext4fs.c | 22 +++++++ fs/fat/fat.c | 17 +++++- fs/fs.c | 130 +++++++++++++++++++++++++++++++++++++++--- fs/ubifs/ubifs.c | 13 +++-- include/btrfs.h | 1 + include/erofs.h | 1 + include/ext4fs.h | 1 + include/fat.h | 3 +- include/ubifs_uboot.h | 1 + 13 files changed, 212 insertions(+), 106 deletions(-) -- 2.37.0 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AD235C43334 for ; Tue, 26 Jul 2022 05:22:57 +0000 (UTC) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4LsQKX0p4sz3bsD for ; Tue, 26 Jul 2022 15:22:56 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lists.ozlabs.org; s=201707; t=1658812976; bh=N5vMrlZsznWTxDDP5VjzMo40WkvisIyfwBvd8HxY+Lw=; h=To:Subject:Date:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:Cc:From; b=F+QVRgnmt5dQiiX0WzitGQb0bPwJPNg3lHPqmQu6fEBzAmFCIgdI7f+FQlAOYW7CE DvG11nfoH4cW3ebPIaP3W4CQ3mA4nG+rPwyoxLYkrBpC5JSxroLScwpFJ8d8gyyWQu iZXQA9cDAjYEpVsxYIsAI8GLSzqNTGlmBpRfe7kWMUISORnwq5+oJcPotD/iuh+SGG 11lqQlsBKYkrmcWDWgLBt3FT7UsfoIBMTOUZw8K/AuI7MabeR0j7SFQxVQZUzEN/Fx 58IsaxB8ckfql4CZx8/DisR28+li4YAbT1YE9Sbr4U9rg7iM5tvDoTQuB800lKYgU+ fJLjjeis2RPdg== Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=suse.com (client-ip=195.135.220.29; helo=smtp-out2.suse.de; envelope-from=wqu@suse.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=suse.com header.i=@suse.com header.a=rsa-sha256 header.s=susede1 header.b=UtLL8nr+; dkim-atps=neutral Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4LsQKP3P4Zz2ywl for ; Tue, 26 Jul 2022 15:22:49 +1000 (AEST) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 7CD311F9C3; Tue, 26 Jul 2022 05:22:37 +0000 (UTC) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 7B8D013A12; Tue, 26 Jul 2022 05:22:35 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 5Ep5Ext632IFOwAAMHmgww (envelope-from ); Tue, 26 Jul 2022 05:22:35 +0000 To: u-boot@lists.denx.de Subject: [PATCH v2 0/8] U-boot: fs: add generic unaligned read offset handling Date: Tue, 26 Jul 2022 13:22:08 +0800 Message-Id: X-Mailer: git-send-email 2.37.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: linux-erofs@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Development of Linux EROFS file system List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Qu Wenruo via Linux-erofs Reply-To: Qu Wenruo Cc: trini@konsulko.com, joaomarcos.costa@bootlin.com, marek.behun@nic.cz, thomas.petazzoni@bootlin.com, miquel.raynal@bootlin.com, linux-erofs@lists.ozlabs.org, linux-btrfs@vger.kernel.org Errors-To: linux-erofs-bounces+linux-erofs=archiver.kernel.org@lists.ozlabs.org Sender: "Linux-erofs" [CHANGELOG] v2->v1: - Fix a linkage error where (U64 % U32) is called without proper helper Fix it with U64 & (U32 - 1), as the U32 value (@blocksize) should always be power of 2, thus (@blocksize - 1) is the mask we want to calculate the offset inside the block. Above change only affects the 4th patch, everything else is not touched. RFC->v1: - More (manual) testing Unfortunately, in the latest master (75967970850a), the fs-tests.sh always seems to hang at preparing the fs image. Thus still has to do manual testing, tested btrfs, ext4 and fat, with aligned and unaligned read, also added soft link read, all looks fine here. Extra testing is still appreciated. - Two more btrfs specific bug fixes All exposed during manual tests - Remove the tailing unaligned block handling In fact, all fses can easily handle such case, just a min() call is enough. - Remove the support for sandboxfs Since it's using read() calls, really no need to do block alignment check. - Enhanced blocksize check Ensure the returned blocksize is not only non-error, but also non-zero. This patchset can be fetched from github: https://github.com/adam900710/u-boot/tree/fs_unaligned_read [BACKGROUND] Unlike FUSE/Kernel which always pass aligned read range, U-boot fs code just pass the request range to underlying fses. Under most case, this works fine, as U-boot only really needs to read the whole file (aka, 0 for both offset and len, len will be later determined using file size). But if some advanced user/script wants to extract kernel/initramfs from combined image, we may need to do unaligned read in that case. [ADVANTAGE] This patchset will handle unaligned read range in _fs_read(): - Get blocksize of the underlying fs - Read the leading block contianing the unaligned range The full block will be stored in a local buffer, then only copy the bytes in the unaligned range into the destination buffer. If the first block covers the whole range, we just call it aday. - Read the remaining range which starts at block aligned offset For most common case, which is 0 offset and 0 len, the code will not be changed at all. Just one extra get_blocksize() call, and for FAT/Btrfs/EROFS they all have cached blocksize, thus it takes almost no extra cost. Although for EXT4, it doesn't seem to cache the blocksize globally, thus has to do a path resolve and grab the blocksize. [DISADVANTAGE] The involved problem is: - Extra path resolving All those supported fs may have to do one extra path resolve if the read offset is not aligned. For EXT4, it will do one extra path resolve just to grab the blocksize. For data read which starts at offset 0 (the most common case), it should cause *NO* difference in performance. As the extra handling is only for unaligned offset. The common path is not really modified. [SUPPORTED FSES] - Btrfs (manually tested*) - Ext4 (manually tested) - FAT (manually tested) - Erofs - ubifs (unable to test, due to compile failure) *: Failed to get the test cases run, thus have to go sandbox mode, and attach an image with target fs, load the target file (with unaligned range) and compare the result using md5sum. For EXT4/FAT, they may need extra cleanup, as their existing unaligned range handling is no longer needed anymore, cleaning them up should free more code lines than the added one. Just not confident enough to modify them all by myself. [UNSUPPORTED FSES] - Squashfs They don't support non-zero offset, thus it can not handle the block aligned range. Need extra help to add block aligned offset support. - Semihostfs - Sandboxfs They all use read() directly, no need to do alignment check at all. Extra testing/feedback is always appreciated. Qu Wenruo (8): fs: fat: unexport file_fat_read_at() fs: btrfs: fix a bug which no data get read if the length is not 0 fs: btrfs: fix a crash if specified range is beyond file size fs: btrfs: move the unaligned read code to _fs_read() for btrfs fs: ext4: rely on _fs_read() to handle leading unaligned block read fs: fat: rely on higher layer to get block aligned read range fs: ubifs: rely on higher layer to do unaligned read fs: erofs: add unaligned read range handling fs/btrfs/btrfs.c | 33 ++++++++--- fs/btrfs/inode.c | 89 +++-------------------------- fs/erofs/internal.h | 1 + fs/erofs/super.c | 6 ++ fs/ext4/ext4fs.c | 22 +++++++ fs/fat/fat.c | 17 +++++- fs/fs.c | 130 +++++++++++++++++++++++++++++++++++++++--- fs/ubifs/ubifs.c | 13 +++-- include/btrfs.h | 1 + include/erofs.h | 1 + include/ext4fs.h | 1 + include/fat.h | 3 +- include/ubifs_uboot.h | 1 + 13 files changed, 212 insertions(+), 106 deletions(-) -- 2.37.0