From mboxrd@z Thu Jan 1 00:00:00 1970 From: Tom Marshall Subject: [RFC] Per-file compression Date: Fri, 17 Apr 2015 15:20:35 -0700 Message-ID: <55318733.4000503@cyngn.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit To: linux-fsdevel@vger.kernel.org Return-path: Received: from mail-pd0-f178.google.com ([209.85.192.178]:35330 "EHLO mail-pd0-f178.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751966AbbDQWUj (ORCPT ); Fri, 17 Apr 2015 18:20:39 -0400 Received: by pdbqd1 with SMTP id qd1so141167394pdb.2 for ; Fri, 17 Apr 2015 15:20:38 -0700 (PDT) Received: from [10.40.0.44] (70-90-190-81-WA.hfc.comcastbusiness.net. [70.90.190.81]) by mx.google.com with ESMTPSA id ms7sm11108261pdb.11.2015.04.17.15.20.37 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 17 Apr 2015 15:20:37 -0700 (PDT) Sender: linux-fsdevel-owner@vger.kernel.org List-ID: I'd like to get some thoughts on whether having transparent per-file compression implemented in the kernel would be desirable. Details: We are running into space issues when upgrading Android devices to the latest version. This issue is particularly troublesome for 64-bit devices that initially shipped without 64-bit support, as the multi-lib files nearly double the space requirements. As it happens, many system files are quite compressible -- namely shared libs and apk files. So implementing filesystem compression is a natural solution. After investigating several well known compression solutions, we did not find any that met our goals: * Compression should be transparent to user space. It would be possible to identify subsets of the filesystem that compress well and handle compression in user space. However, doing this involves repeating similar work in different areas of code, and can never fully utilize compression across the entire filesystem. * Compression must work with existing Android filesystems. There are filesystems that have native compression. However, switching to a new filesystem on a released device is not really desirable. * Compression should be implemented as transparently as possible. cloop is filesystem independent, but it makes the underlying device read-only. This could be mitigated somewhat with overlayfs, but that would be a rather complex solution. * Compression should be independent of the filesystem. e2compr may work for ext2, but it seems abandoned and does not address other filesystem types. So, I wrote a thing called 'zfile' that hooks into the VFS layer and intercepts file_operations to do file (de)compression on the fly. When a file is opened, it reads and decompresses the data into memory. The file may be read, written, and mmaped in the usual way. If the contents are changed, the data is compressed and written back. A working patch for an older kernel version may be found at: http://review.cyanogenmod.org/#/c/95220/ Note this is mostly just a prototype at the moment. I'm fully aware that it has some bugs and limitations. Pretty major ones, I'm sure. However, if this is something that is generally desirable, I would look forward to working with the VFS maintainers to make it merge worthy. Thanks!