From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ig0-f177.google.com ([209.85.213.177]:48377 "EHLO mail-ig0-f177.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751303AbaHAOQL (ORCPT ); Fri, 1 Aug 2014 10:16:11 -0400 Received: by mail-ig0-f177.google.com with SMTP id hn18so1622969igb.10 for ; Fri, 01 Aug 2014 07:16:09 -0700 (PDT) Message-ID: <53DBA128.8060605@gmail.com> Date: Fri, 01 Aug 2014 10:16:08 -0400 From: Austin S Hemmelgarn MIME-Version: 1.0 To: dsterba@suse.cz, Timofey Titovets , linux-btrfs@vger.kernel.org Subject: Re: Btrfs offline deduplication References: <53DB6948.3000009@gmail.com> <20140801132308.GF1553@twin.jikos.cz> In-Reply-To: <20140801132308.GF1553@twin.jikos.cz> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms050305080402020307030809" Sender: linux-btrfs-owner@vger.kernel.org List-ID: This is a cryptographically signed message in MIME format. --------------ms050305080402020307030809 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 2014-08-01 09:23, David Sterba wrote: > On Fri, Aug 01, 2014 at 06:17:44AM -0400, Austin S Hemmelgarn wrote: >> I do think however that having the option of a background thread doing= >> deduplication asynchronously is a good idea, but then you would have t= o >> have some way to trigger it on individual files/trees, and triggering = on >> writes like the autodefrag thread does doesn't make much sense. Havin= g >> some userspace program to tell it to run on a given set of files would= >> probably be the best approach for a trigger. I don't remember if this= >> kind of thing was also included in the online deduplication patches th= at >> got posted a while back or not. >=20 > IIRC the proposed implementation only merged new writes with existing > data. >=20 > For the out-of-band ("off-line") dedup there's bedup > (https://github.com/g2p/bedup) or Mark's duperemove tool > (https://github.com/markfasheh/duperemove) that work on a set of files.= >=20 Something kernel-side to do the work asynchronously would be nice, especially if it could leverage the check-sums that BTRFS already stores for the blocks. Having a userspace interface for offline deduplication similar to that for scrub operations would even better. --------------ms050305080402020307030809 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIGuDCC BrQwggScoAMCAQICAw8BRDANBgkqhkiG9w0BAQ0FADB5MRAwDgYDVQQKEwdSb290IENBMR4w HAYDVQQLExVodHRwOi8vd3d3LmNhY2VydC5vcmcxIjAgBgNVBAMTGUNBIENlcnQgU2lnbmlu ZyBBdXRob3JpdHkxITAfBgkqhkiG9w0BCQEWEnN1cHBvcnRAY2FjZXJ0Lm9yZzAeFw0xNDA1 MTIxNDEwMzJaFw0xNDExMDgxNDEwMzJaMGMxGDAWBgNVBAMTD0NBY2VydCBXb1QgVXNlcjEj MCEGCSqGSIb3DQEJARYUYWhmZXJyb2luN0BnbWFpbC5jb20xIjAgBgkqhkiG9w0BCQEWE2Fo ZW1tZWxnQG9oaW9ndC5jb20wggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDbLUaL Gs4JTdU7sgr0MzD57CMUAv307ddC9pxooDMN3PiUvzEd5kLtBCh8KDB1wbMdfm4hte2rDd+j hM1tIq67BvNbdDPztOcBZwT2/3OVyyG4B1ddCqUyt03zGKw6Y34eHNfapsZiiItX0GBNfjHU Wv+WDo+XNha/WmGSSMv21HkftF9XA1KC9Bpr9JJI23MKK7T2g/7b3KoGZlx3ekLIJsF5B7+B DMPPDqOHQbRnccyOHEMyhM13g6WoAbU+3aKYc+C/9UsYtDV+xlvBLWagky1acstD5wOA35V6 uDRbUhD+vOjuMRMCj9jJOIYqa6AeSagBjxRnisJr0RFzQ4f+NjGCHPaFTvRvbkiXh4q22doT 0SxbNBUm7B9ANugIOtS9/VQhTWKDi//WTqZQ7Ecl4yVJbMCUg/iaRHMCGS41vqMICPszRidW rL04NwS9D2cREEY1y/xrNo0ZvKPZu6tLhxhPf7w+5rsN3+wWxGaR1hNpnVUT9AeacLKZO6W9 FsRT3Unkr91IhQATHTKYr4EAkjN/5lgvA+sxp5TxxsUnoJYrD8IHf8aYfJsAHMleBwx4xSeZ tw/n5iIjJjFZq9IRZ1zQhK62p+a5vJ2vlJHjTgavhQrfb1pUOjbqsnI4ndQ5hNosL9el4Kxq Yko+HsxVEmSwSsjq6cV2L3oz0z8NUwIDAQABo4IBWTCCAVUwDAYDVR0TAQH/BAIwADBWBglg hkgBhvhCAQ0ESRZHVG8gZ2V0IHlvdXIgb3duIGNlcnRpZmljYXRlIGZvciBGUkVFIGhlYWQg b3ZlciB0byBodHRwOi8vd3d3LkNBY2VydC5vcmcwDgYDVR0PAQH/BAQDAgOoMEAGA1UdJQQ5 MDcGCCsGAQUFBwMEBggrBgEFBQcDAgYKKwYBBAGCNwoDBAYKKwYBBAGCNwoDAwYJYIZIAYb4 QgQBMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcwAYYWaHR0cDovL29jc3AuY2FjZXJ0Lm9y ZzAxBgNVHR8EKjAoMCagJKAihiBodHRwOi8vY3JsLmNhY2VydC5vcmcvcmV2b2tlLmNybDA0 BgNVHREELTArgRRhaGZlcnJvaW43QGdtYWlsLmNvbYETYWhlbW1lbGdAb2hpb2d0LmNvbTAN BgkqhkiG9w0BAQ0FAAOCAgEAIokFPcW8+cO2Clu0Ei+ehAmQRBHfV5RWJ8aMVLXOCfiJX0ch IjVSIt6I3uQaR4J1ZIAjCSPkbpfZQDaLoGFI5j8aYEQhOeKxrvOMzY9/aSUYabCJIhE/sX64 klFV0bzm+PR9cDMWeQ9BoZf0m8UROPSfDnrjEk+p04hGg3pAZMcSwCzxdb604NHjgHJmf2xG UQVzQgC6Ek/BKat0xuPTuPmtPv9OicK75CPmLZKYW3rFpCD6bhb1mm+ROcCNhniRY2LYm9YN QdlHQUzTFqj0tvuYrzNI3LNV4PjEfN8z6omPCT2Rq8/uKLseN+m8F0ioqm+cphqpmzKoDUpN nePLkqDFUFWCeWRxSjBTy4IMVUfdNXriVGihH8hyIICQiOfmmBOzhzUifdomJuTGtoXRuHVT R2f/YdrJrLnKI4f+Othdp7F3KhB4c6JiOnTEH5J8n9q3rFjt4MPRwcjIHMhmF5nZVQlgxEMo 1cPCmvG1D9tcgXbH79jjqydo9SDXhzLQob7axkzGRY96IstNcvoQ/UNsdPPfFMYlHtGz4TxT DhBjv4ERskGmKBZrfmxkXkcuTV/gcykct6Xvw9YXb8WTL4qSYHSYk9fReVLgE/L4RBUpX2JJ QvIR0AJLER165/aZlQXZtuJjnfxJtJTJZZ+Gor9h0G2kuR5Dy0JuYdBO4t4xggShMIIEnQIB ATCBgDB5MRAwDgYDVQQKEwdSb290IENBMR4wHAYDVQQLExVodHRwOi8vd3d3LmNhY2VydC5v cmcxIjAgBgNVBAMTGUNBIENlcnQgU2lnbmluZyBBdXRob3JpdHkxITAfBgkqhkiG9w0BCQEW EnN1cHBvcnRAY2FjZXJ0Lm9yZwIDDwFEMAkGBSsOAwIaBQCgggH1MBgGCSqGSIb3DQEJAzEL BgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTE0MDgwMTE0MTYwOFowIwYJKoZIhvcNAQkE MRYEFCOuB/F03H7rdPmV3lFkEzrqsj94MGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEq MAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwIC AUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwgZEGCSsGAQQBgjcQBDGBgzCBgDB5MRAwDgYD VQQKEwdSb290IENBMR4wHAYDVQQLExVodHRwOi8vd3d3LmNhY2VydC5vcmcxIjAgBgNVBAMT GUNBIENlcnQgU2lnbmluZyBBdXRob3JpdHkxITAfBgkqhkiG9w0BCQEWEnN1cHBvcnRAY2Fj ZXJ0Lm9yZwIDDwFEMIGTBgsqhkiG9w0BCRACCzGBg6CBgDB5MRAwDgYDVQQKEwdSb290IENB MR4wHAYDVQQLExVodHRwOi8vd3d3LmNhY2VydC5vcmcxIjAgBgNVBAMTGUNBIENlcnQgU2ln bmluZyBBdXRob3JpdHkxITAfBgkqhkiG9w0BCQEWEnN1cHBvcnRAY2FjZXJ0Lm9yZwIDDwFE MA0GCSqGSIb3DQEBAQUABIICAImLK8XWn8jPR2QkKwMDsZ75V9MpDkXTUc3Qk5R18IUGQ8ti s4YTgHclCkiDaoKOSZznPS6MgFxzHVoOUaGh/gmttMZgfJYB2iRBrZ9Ov6vCEYmK2w+9fLuQ +NwgswnydQ3wdJ2NJvq1280Zdyycq2EoM/rv2UEs3z53UOivagUakvW4U1VcABBERpE8rbeR I9DRyUG8Fu0hD4YKRoST/0D5wm2JTEuUDUalr1iw6DhgKgTNkW2LlXzXbofU2QELOk4mMXt9 z3OIx0DEINuLBesOQqw83Rnnqz0yWJHlcg0jah7oHQq/aWwQXi6FqIwDAdepo8GdbxVhWssF B/pEqzfERuptBQgoZodlV9nwo0/h20UhmtVP6R4TIXGtCqv1HaVHX7TIhcFFV8TT0iPPZMpj Q7JMLGxlk0BLMlUDeLSJ+BJwJOiW55rWgDsLhCl6Ld+pFwGEqfYt7cUzn5pfxU0AO2023mVu GY2n+zeiESnT5YjyuHQXga6M14E+A5p/5aV3O9xAbO8Z60wn2XQEDfFnORKk61Mqday2zNvu 0ccaEaNmN5QhFne/5QQrzyFzxycmu1ZSwrmf9n33ib5WcwpKLUBsvy0bL38dSz4Ms6UdHhVn c9GVrKZEyfiPJvTKMQ1eL5BS0qY2eMnlnRbif2z2pFGuqKhkN3JrmOpRuM60AAAAAAAA --------------ms050305080402020307030809--