joining to contribute

joining to contribute

[Anthony Riley]

Hey folks,

I thought I would finally take a swing at things I've wanted to be an
kernel/fs dev fora few years now. My current $job is as an
Infrastructure Engineer. I'm currently teaching myself C and have
background in shell scripting & python. I love doing deep dives and
learning about linux internals.  I've read the btrfs.wiki and can't
really decide which project to choose to start.

Also should I go through this https://kernelnewbies.org/FirstKernelPatch first?
Or should i start with something in Userspace?

Thanks in advance,
Really eager to get going.

Anthony
-- 
Anthony Riley II
(415)407.9687
Linux user  #521929
http://counter.li.org/!

Re: joining to contribute

[Qu Wenruo]

On 2017年09月01日 11:36, Anthony Riley wrote:
> Hey folks,
> 
> I thought I would finally take a swing at things I've wanted to be an
> kernel/fs dev fora few years now. My current $job is as an
> Infrastructure Engineer. I'm currently teaching myself C and have
> background in shell scripting & python. I love doing deep dives and
> learning about linux internals.  I've read the btrfs.wiki and can't
> really decide which project to choose to start.
> 
> Also should I go through this https://kernelnewbies.org/FirstKernelPatch first?
> Or should i start with something in Userspace?

Well, personally I strongly recommended to start with btrfs on-disk 
format first, and then btrfs-progs/test cases, and kernel contribution 
as final objective.

BTW, if you want to start with btrfs on-disk format, print-tree.c from 
btrfs-progs is a good start point and btrfs wiki has relatively well 
documented entry for it.
https://btrfs.wiki.kernel.org/index.php/Btrfs_design
https://btrfs.wiki.kernel.org/index.php/Btree_Items

Thanks,
Qu
> 
> Thanks in advance,
> Really eager to get going.
> 
> Anthony
> 

Re: joining to contribute

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 3418 bytes --]

On Fri, Sep 01, 2017 at 01:15:45PM +0800, Qu Wenruo wrote:
> On 2017年09月01日 11:36, Anthony Riley wrote:
> >Hey folks,
> >
> >I thought I would finally take a swing at things I've wanted to be an
> >kernel/fs dev fora few years now. My current $job is as an
> >Infrastructure Engineer. I'm currently teaching myself C and have
> >background in shell scripting & python. I love doing deep dives and
> >learning about linux internals.  I've read the btrfs.wiki and can't
> >really decide which project to choose to start.
> >
> >Also should I go through this https://kernelnewbies.org/FirstKernelPatch first?
> >Or should i start with something in Userspace?
> 
> Well, personally I strongly recommended to start with btrfs on-disk
> format first, and then btrfs-progs/test cases, and kernel
> contribution as final objective.

   Pick a project which bothers you -- is there some feature that you
want to have, or a particular bug, or a sharp corner that needs
rounding off?

   This next bit is purely my opinion. Feel free to ignore it if it
doesn't float your boat.

   For a relatively easy example (from a code point of view), there's
a bug with send-receive where stuff breaks if you try snapshotting and
then sending a subvolume which already has a received-uuid set.
There's probably several ways of dealing with this.

   Alongside this, there's also a requirement for being able to do
round-trip send/receive while preserving the ability to do incremental
sends. This is likely to be related to the above bug-fix. I did a
complete write-up of what's happening, and what needs to happen, here:

http://www.spinics.net/lists/linux-btrfs/msg44089.html

   If you can fix the first bug in a way that doesn't rule out the
round-trip work, that's great. If you can also get the round-trip
stuff working, that's even better (but be warned that it will get
postponed until kdave is ready for all the stream format changes to
happen at once).

> BTW, if you want to start with btrfs on-disk format, print-tree.c
> from btrfs-progs is a good start point and btrfs wiki has relatively
> well documented entry for it.

   Seconded on this. For the on-disk format, it's also useful to
create a small test filesystem and use btrfs-debug-tree on it as you
do stuff to it (create files, create subvols, make links, modify
files). Do this in conjunction with the "Data Structures" page, and
you can see how it all actually fits together.

   It took me a couple of weeks to learn how it all worked the first
time round, but I didn't have much detailed documentation to work from
back then.

   Since you speak python, there's also Hans's python-btrfs:

https://github.com/knorrie/python-btrfs

> https://btrfs.wiki.kernel.org/index.php/Btrfs_design
> https://btrfs.wiki.kernel.org/index.php/Btree_Items

And, more generally: https://btrfs.wiki.kernel.org/index.php/Main_Page#Developer_documentation

I'd also point out the Data Structures and Trees pages linked
there. Some of the information is a bit out of date, or represents a
prototype of what it's describing. The source code is canonical -- use
the documentation as a guide to help you see where in the source to
look.

   Hugo.

-- 
Hugo Mills             | I'm always right. But I might be wrong about that.
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: joining to contribute

[Hans van Kranenburg]

On 09/01/2017 07:15 AM, Qu Wenruo wrote:
> 
> 
> On 2017年09月01日 11:36, Anthony Riley wrote:
>> Hey folks,
>>
>> I thought I would finally take a swing at things I've wanted to be an
>> kernel/fs dev fora few years now. My current $job is as an
>> Infrastructure Engineer. I'm currently teaching myself C and have
>> background in shell scripting & python. I love doing deep dives and
>> learning about linux internals.  I've read the btrfs.wiki and can't
>> really decide which project to choose to start.
>>
>> Also should I go through this
>> https://kernelnewbies.org/FirstKernelPatch first?
>> Or should i start with something in Userspace?
> 
> Well, personally I strongly recommended to start with btrfs on-disk
> format first, and then btrfs-progs/test cases, and kernel contribution
> as final objective.
> 
> BTW, if you want to start with btrfs on-disk format, print-tree.c from
> btrfs-progs is a good start point and btrfs wiki has relatively well
> documented entry for it.
> https://btrfs.wiki.kernel.org/index.php/Btrfs_design
> https://btrfs.wiki.kernel.org/index.php/Btree_Items
> 

(Insert THX sound effect here)

Previous very useful Qu answer:

https://www.spinics.net/lists/linux-btrfs/msg66489.html

-- 
Hans van Kranenburg

Re: joining to contribute

[Piotr Pawłow]

Hello,
>
>    Alongside this, there's also a requirement for being able to do
> round-trip send/receive while preserving the ability to do incremental
> sends. This is likely to be related to the above bug-fix. I did a
> complete write-up of what's happening, and what needs to happen, here:
>
> http://www.spinics.net/lists/linux-btrfs/msg44089.html

I missed that discussion, but I proposed a different solution in a similar thread about send/receive (https://www.spinics.net/lists/linux-btrfs/msg60694.html)

I think it's not very useful that received_uuid encodes where the subvolume comes from. All that send / receive should care about, is that the contents of source(s) used for incremental send match the contents of subvolumes on the receive side. Let's call it, for example, "contents_uuid". The rules would be simple: operations that preserve contents preserve contents_uuid ; operations that change contents change contents_uuid. For simplicity and performance reasons, in order to not need tracking of changes, we could allow for some false positives, where contents_uuid changed when data did not. A simpler to implement set of rules could look like this:

- rw subvolumes have no contents_uuid
- changing rw subvolume to ro assigns a random contents_uuid
- ro snapshot of rw subvolume gets a random contents_uuid
- ro snapshot of ro subvolume preserves contents_uuid
- send/receive preserves contents_uuid (after successful receive)

And then the rule for send / receive would be:
- send transmits contents_uuid of subvolumes used as clone sources, which are matched to subvolumes having identical contents_uuid on the receive side.

Does it make sense? Did I miss something? I haven't received any feedback last time, which is why I bring it up again for discussion.