* [PATCH 2/2] btrfs-progs: introduce kernelsrc directory and script to update it
2013-02-12 8:37 [RFC][PATCH 0/2] btrfs-progs: introduce kernelsrc directory to update it Filipe Brandenburger
2013-02-12 8:37 ` [PATCH 1/2] btrfs-progs: anchor gitignore patterns on the root of the tree Filipe Brandenburger
@ 2013-02-12 8:37 ` Filipe Brandenburger
1 sibling, 0 replies; 3+ messages in thread
From: Filipe Brandenburger @ 2013-02-12 8:37 UTC (permalink / raw)
To: linux-btrfs; +Cc: Filipe Brandenburger
The kernelsrc directory will contain a copy of the Btrfs-related files from the
Linux kernel source tree (mainly the files in fs/btrfs/ directory.)
This commit introduces an initial filelist (generated by comparing the list of
files in both directories) and a script to fetch the files from a local clone of
the kernel git directory into the kernelsrc directory. When updating the
directory, a git tag needs to be specified, which is recorded in a local file,
in a way that it is possible to track down to which kernel release the kernelsrc
files originally belong.
A README file was also added to explain the idea behind it and to document the
procedure to update the kernelsrc directory.
Signed-off-by: Filipe Brandenburger <filbranden@google.com>
---
kernelsrc/README.kernelsrc | 108 ++++++++++++++++++++++++
kernelsrc/update_kernelsrc.filelist | 23 ++++++
kernelsrc/update_kernelsrc.py | 158 ++++++++++++++++++++++++++++++++++++
3 files changed, 289 insertions(+)
create mode 100644 kernelsrc/README.kernelsrc
create mode 100644 kernelsrc/update_kernelsrc.filelist
create mode 100755 kernelsrc/update_kernelsrc.py
diff --git a/kernelsrc/README.kernelsrc b/kernelsrc/README.kernelsrc
new file mode 100644
index 0000000..0aed4bb
--- /dev/null
+++ b/kernelsrc/README.kernelsrc
@@ -0,0 +1,108 @@
+
+The kernelsrc directory tree
+============================
+
+The `kernelsrc' directory in the btrfs-progs tree contains a copy of some of the
+files in the Linux kernel tree, mainly the files in fs/btrfs/, that are shared
+between the implementation of the filesystem kernel module and some of the
+command-line utilities.
+
+These files should never be edited on the btrfs-progs tree. Changes to these
+files should be committed and pushed to the Linux kernel git tree (optionally
+using #define's and #ifdef's to make code compatible with both kernel and user
+spaces) and then it should be updated here from the kernel tree.
+
+In order to update the `kernelsrc' directory, you need to have a checkout of
+both the btrfs-progs git tree and the Linux kernel git tree.
+
+Additionally to the kernel source files, the `kernelsrc' directory tree also
+contains the scripts and control files used to update this same tree.
+
+
+Updating the kernelsrc tree with update_kernelsrc.py
+====================================================
+
+In order to update the `kernelsrc' tree to include the source code files from a
+newer version of the Linux kernel tree, these are the steps that should be
+followed:
+
+ 1. Make sure that you are in the root of the btrfs-progs git tree and that
+ the tree is clean (i.e., run `git status' and check its output.)
+
+ 2. Make sure you have a clone of the Linux kernel git tree. Let's assume
+ for this example that your kernel git tree is in directory ../linux/
+
+ 3. Make sure you have know to which tag of the Linux kernel you want to
+ upgrade the `kernelsrc' tree to (e.g. v3.7 or v3.8-rc7)
+
+ 4. (Optional:) You may change the list of files that will be copied from
+ the Linux kernel tree. In order to do that, edit the file
+ `kernelsrc/update_kernelsrc.filelist' and either add or remove entries
+ from that file.
+
+ 5. Run the following command from the root of the btrfs-progs git tree:
+
+ $ kernelsrc/update_kernelsrc.py ../linux/ v3.7
+
+ (assuming Linux kernel git tree at ../linux/ and upgrading to tag v3.7)
+
+ 6. Check whether the update matches your expectations. In particular, the
+ commands `git status' and `git diff --cached' may be helpful.
+
+ 7. Build btrfs-progs (using `make') and test them to make sure they are
+ still compatible with the new kernel source tree. Any needed adjustments
+ may be done by editing files *OUTSIDE* of the `kernelsrc' tree. Once
+ everything seems to be in place and working correctly, you may add those
+ files to the same git commit using the `git add' command.
+
+ 8. Commit the change, using the `git commit' command. You may use the
+ suggested commit command from the script output, with a standard commit
+ message. The editor will open anyways, so you can add more details
+ (particularly about files outside the `kernelsrc' tree that needed
+ changes.)
+
+
+Migration to kernelsrc
+======================
+
+Using the `kernelsrc' tree allows for a gradual migration from the current
+situation, where copies of the original source files from the kernel tree were
+made in btrfs-progs, adjustments were made to allow them to compile in user
+space, and they diverged further over time.
+
+First, it allows checking out the files from the kernel tree and adding them to
+the btrfs-progs without actually using them initially. Then, it allows gradually
+migrating the files one by one. One way to do it would be replacing the file in
+the btrfs-progs root directory with a symbolic link to the same file in the
+`kernelsrc' tree. Another way would be integrating the build of the `kernelsrc'
+files into the Makefile (assuming all of them compile in userspace without
+errors) so that they get built and can be linked from there. The integration may
+be switched from the former to the latter at any point in a commit that
+basically only touches the Makefiles and symlinks.
+
+
+Alternatives to using a script to update kernelsrc
+==================================================
+
+When coming up with the idea for `update_kernelsrc.py', my first idea was to use
+`git submodule' instead. It seems to perfectly capture the idea of having a
+repository with source code files that are used by a project in another
+repository. So, in theory, it would be possible to have the Linux kernel git
+tree as a submodule of the btrfs-progs source tree (it could even be placed
+under a `kernelsrc' subdirectory.) Git would then track the revision of the
+Linux git tree, and it would be possible to upgrade the tree by doing a checkout
+on the Linux tree and then committing on the btrfs-progs tree, which would
+simply record the git revision where the Linux tree was pointing to at the time
+of the commit.
+
+The main problem with this approach is the size of the Linux git tree. A clone
+of the btrfs-progs tree takes about 2 or 3 MB of diskspace, while the Linux tree
+takes about 1 GB. I tried to look at options of doing a `sparse checkout' in
+git, but it doesn't seem to be possible to do it for a submodule... Even if it
+was, I'm not fully convinced that it would reduce the diskspace requirements, as
+it would probably still need to fetch the objects into the .git tree anyways.
+
+So, the idea of a script that would update the tree by copying the files and
+keeping the revision in control files seemed like a close enough solution that
+would implement a similar concept without the diskspace problems.
+
diff --git a/kernelsrc/update_kernelsrc.filelist b/kernelsrc/update_kernelsrc.filelist
new file mode 100644
index 0000000..50cdf95
--- /dev/null
+++ b/kernelsrc/update_kernelsrc.filelist
@@ -0,0 +1,23 @@
+fs/btrfs/ctree.c
+fs/btrfs/ctree.h
+fs/btrfs/dir-item.c
+fs/btrfs/disk-io.c
+fs/btrfs/disk-io.h
+fs/btrfs/extent_io.c
+fs/btrfs/extent_io.h
+fs/btrfs/extent-tree.c
+fs/btrfs/file-item.c
+fs/btrfs/hash.h
+fs/btrfs/inode-item.c
+fs/btrfs/inode-map.c
+fs/btrfs/ioctl.h
+fs/btrfs/print-tree.c
+fs/btrfs/print-tree.h
+fs/btrfs/qgroup.c
+fs/btrfs/root-tree.c
+fs/btrfs/send.h
+fs/btrfs/transaction.h
+fs/btrfs/version.h
+fs/btrfs/volumes.c
+fs/btrfs/volumes.h
+include/linux/list.h
diff --git a/kernelsrc/update_kernelsrc.py b/kernelsrc/update_kernelsrc.py
new file mode 100755
index 0000000..0608671
--- /dev/null
+++ b/kernelsrc/update_kernelsrc.py
@@ -0,0 +1,158 @@
+#! /usr/bin/python
+
+import os
+import subprocess
+import sys
+
+SCRIPT_NAME = 'kernelsrc/update_kernelsrc.py'
+KERNELSRC_GIT_DIR = None
+KERNELSRC_WORK_TREE = 'kernelsrc/'
+
+# read filelist stripping comments and blank lines
+def read_filelist():
+ with open('kernelsrc/update_kernelsrc.filelist') as f:
+ filelist = [l for l in (l.split('#')[0].strip() for l in f.readlines()) if l]
+ if filelist:
+ return filelist
+ else:
+ raise Exception('update_kernelsrc.filelist does not contain any files')
+
+# checkout files from kernel git tree
+def checkout_kernelsrc(rev, filelist):
+ cmdline = ['git', '--git-dir=' + KERNELSRC_GIT_DIR, '--work-tree=' + KERNELSRC_WORK_TREE,
+ 'checkout', rev, '--'] + filelist
+ rc = subprocess.call(cmdline)
+ if rc != 0:
+ raise Exception('git checkout from kernel source git repo returned %d' % rc)
+
+# run a git add on the updated kernelsrc tree to add the changes to the index
+def add_kernelsrc_to_index():
+ # add them to git index of this git tree
+ rc = subprocess.call(['git', 'add', KERNELSRC_WORK_TREE])
+ if rc != 0:
+ raise Exception('git add %s returned %d' % (KERNELSRC_WORK_TREE, rc))
+
+# remove current files from kernelsrc to prepare to receive new ones
+def cleanup_kernelsrc():
+ for dirpath, dirnames, filenames in os.walk(KERNELSRC_WORK_TREE):
+ for f in filenames:
+ # ignore files named update_kernelsrc.* and README* on root of the kernelsrc/ tree:
+ if (dirpath == KERNELSRC_WORK_TREE) and (f.startswith('update_kernelsrc.') or f.startswith('README')):
+ continue
+ # otherwise, git rm it
+ filepath = os.path.join(dirpath, f)
+ rc = subprocess.call(['git', 'rm', '-q', filepath])
+ if rc != 0:
+ raise Exception('git rm %s returned %d' % (filepath, rc))
+ # second pass: remove directories, use a stack to remove them in the right order
+ stack = []
+ for dirpath, dirnames, filenames in os.walk(KERNELSRC_WORK_TREE):
+ if dirpath != KERNELSRC_WORK_TREE:
+ stack.append(dirpath)
+ for d in reversed(stack):
+ os.rmdir(d)
+
+# check whether current tree is clean and ready to receive update
+# only allowed change in index is to kernelsrc/update_kernelsrc.filelist
+# changes not in index must be outside of kernelsrc tree
+def check_for_clean_tree():
+ status = subprocess.check_output(['git', 'status', '--porcelain']).splitlines()
+ for l in status:
+ worktree = l[0]
+ index = l[1]
+ path = l[3:]
+ if (path == os.path.join(KERNELSRC_WORK_TREE, 'update_kernelsrc.filelist')) and (worktree in ' M') and (index in ' M'):
+ # accept changes to the filelist in index and/or worktree
+ continue
+ if (not path.startswith(KERNELSRC_WORK_TREE)) and (worktree == '?') and (index == '?'):
+ # accept files with unknown status outside of kernelsrc
+ continue
+ # otherwise, abort
+ raise Exception('current tree is not clean! (e.g. %s)' % path)
+
+# resolve tag into revision
+def resolve_tag(tag):
+ cmdline = ['git', '--git-dir=' + KERNELSRC_GIT_DIR, 'rev-parse', tag + '^0']
+ rev = subprocess.check_output(cmdline).strip()
+ return rev
+
+# check whether the new tag is an upgrade from the previous one
+# in other words, if the previous tag is an ancestor of the new one
+def check_upgrade(rev):
+ # previous revision stored in kernelsrc/update_kernelsrc.rev
+ oldrevpath = os.path.join(KERNELSRC_WORK_TREE, 'update_kernelsrc.rev')
+ if not os.path.exists(oldrevpath):
+ # if it doesn't exist, then just assume it's OK for an upgrade
+ # but return None in order to signal the this is the first checkout
+ return None
+ with open(oldrevpath) as f:
+ oldrev = f.read().strip()
+ # use git merge-base to check what's the merge base between them
+ # if it's an upgrade, it will match oldrev
+ cmdline = ['git', '--git-dir=' + KERNELSRC_GIT_DIR, 'merge-base', oldrev, rev]
+ mergebase = subprocess.check_output(cmdline).strip()
+ if mergebase != oldrev:
+ raise Exception('new revision %s is not a descendent of previous revision %s' % (rev, oldrev))
+ oldtagpath = os.path.join(KERNELSRC_WORK_TREE, 'update_kernelsrc.tag')
+ # return the old tag (for git commit sample message)
+ try:
+ with open(oldtagpath) as f:
+ oldtag = f.read().strip()
+ return oldtag
+ except OSError:
+ return oldrev
+
+# store tag (and revision) for future query/comparison
+def store_tag(tag, rev):
+ with open(os.path.join(KERNELSRC_WORK_TREE, 'update_kernelsrc.tag'), 'w') as f:
+ print >>f, tag
+ with open(os.path.join(KERNELSRC_WORK_TREE, 'update_kernelsrc.rev'), 'w') as f:
+ print >>f, rev
+
+# do it all!
+def main():
+ global KERNELSRC_GIT_DIR
+ if len(sys.argv) != 3:
+ raise Exception('usage: %s <kernelsrc> <tag>' % SCRIPT_NAME)
+ if sys.argv[0] != SCRIPT_NAME:
+ raise Exception('script must be called as %s from root directory in btrfs-progs tree' % SCRIPT_NAME)
+ if not os.path.isdir('.git'):
+ raise Exception('script must be called from root directory in btrfs-progs git tree')
+ if not os.path.isdir(KERNELSRC_WORK_TREE):
+ raise Exception('weird, this looks like a git tree but could not find directory %s' % KERNELSRC_WORK_TREE)
+ if not os.path.isdir(sys.argv[1]):
+ raise Exception('first parameter must be the path to the kernel source directory')
+ KERNELSRC_GIT_DIR = os.path.join(sys.argv[1], '.git')
+ if not os.path.isdir(KERNELSRC_GIT_DIR):
+ raise Exception('first parameter must be the path to the root of the kernel git tree')
+ # all checks so far, let's get the new tag and see if it's good:
+ tag = sys.argv[2]
+ rev = resolve_tag(tag)
+ oldtag = check_upgrade(tag)
+ # all good, let's check if the tree is clean
+ check_for_clean_tree()
+ # let's see if we can get the file list ok
+ filelist = read_filelist()
+ # all good! now let's do the update (remove followed by checkout)
+ cleanup_kernelsrc()
+ checkout_kernelsrc(rev, filelist)
+ # done upgrade, still not in index though...
+ # before adding it to the index, let's record tag and rev
+ store_tag(tag, rev)
+ # and finally, add to git index
+ add_kernelsrc_to_index()
+ # done!
+ print 'Done upgrading kernelsrc to %s.' % tag
+ print 'Please test the new tree and commit when ready.'
+ print ''
+ print 'Use the following command for commit:'
+ if oldtag is None:
+ print ' git commit -em "btrfs-progs: initial checkout of kernelsrc %s"' % tag
+ else:
+ print ' git commit -em "btrfs-progs: upgrade kernelsrc from %s to %s"' % (oldtag, tag)
+ print ''
+
+# if called as a script, just run main
+if __name__ == '__main__':
+ main()
+
--
1.8.1.2
^ permalink raw reply related [flat|nested] 3+ messages in thread