From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F06DEC433C1 for ; Mon, 29 Mar 2021 13:17:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AB13B6195B for ; Mon, 29 Mar 2021 13:17:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231513AbhC2NRM (ORCPT ); Mon, 29 Mar 2021 09:17:12 -0400 Received: from mail-out.m-online.net ([212.18.0.9]:53543 "EHLO mail-out.m-online.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229479AbhC2NQu (ORCPT ); Mon, 29 Mar 2021 09:16:50 -0400 Received: from frontend01.mail.m-online.net (unknown [192.168.8.182]) by mail-out.m-online.net (Postfix) with ESMTP id 4F8Clh66Clz1qt4Z; Mon, 29 Mar 2021 15:16:48 +0200 (CEST) Received: from localhost (dynscan1.mnet-online.de [192.168.6.70]) by mail.m-online.net (Postfix) with ESMTP id 4F8Clh5rhwz1r1MH; Mon, 29 Mar 2021 15:16:48 +0200 (CEST) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.182]) by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new, port 10024) with ESMTP id uyd6MrEA8TPE; Mon, 29 Mar 2021 15:16:48 +0200 (CEST) X-Auth-Info: +xEoWXzYbZSF62p4WF9qugfSb+GtcLH1Amm6dNTa4xo= Received: from [10.88.0.186] (dslb-084-056-254-233.084.056.pools.vodafone-ip.de [84.56.254.233]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.mnet-online.de (Postfix) with ESMTPSA; Mon, 29 Mar 2021 15:16:48 +0200 (CEST) To: linux-btrfs@vger.kernel.org Cc: Henning Schild From: Claudius Heine Subject: btrfs-send format that contains binary diffs Organization: Denx Software Engineering Message-ID: Date: Mon, 29 Mar 2021 15:16:47 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.9.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org Hi, I am currently investigating the possibility to use `btrfs-stream` files (generated by `btrfs send`) for deploying a image based update to systems (probably embedded ones). One of the issues I encountered here is that btrfs-send does not use any diff algorithm on files that have changed from one snapshot to the next. One way to implement this would be to add some sort of 'patch' command to the `btrfs-stream` format. Is this something upstream would be interested in? Lets say we introduce a new `btrfs-send` format, lets call it `btrfs-delta-stream`, which could can be created from a `btrfs-stream`: 1. For all `write` commands, check the requirements: - Does the file already exists in the old snapshot? - Is the file smaller than xMiB (this depends on the diff-algo and the available resources) 2. If the file fulfills those requirements, replace 'write' command with 'patch' command, and calculate the binary delta. Also check if the delta is actually smaller than the data of the new file. Possible add the used binary diff algo as well as a checksum of the 'old' file to the command as well. This file format can of course be converted back to `btrfs-stream` and then applied with `btrfs-receive`. I would probably start with `bsdiff` for the diff algorithm, but maybe we want to be flexible here. Of course if `btrfs-delta-stream` is implemented in `btrfs-progs` then, we can create and apply this format directly. regards, Claudius