Re: Ext2 detailed documentation

[Livio Baldini Soares <livio@ime.usp.br>]

  Hey James,

James Buchanan writes:
> Aside from reading the source code (and fully understanding it, which
> I am not sure I am capable of yet), is there any detailed
> documentation on how the Ext2 fs works?  Specifically, I'd like to
> know some fairly high-level details so I know what to look for when
> I'm studying the sources.
> 
> Here's what I'm looking for:  where's the code that gets blocks of
> data off the disk and sends it up through the kernel via the Ext2 fs
> code?  And vice versa.  I see a lot of code dealing with pages, does
> this mean the ext2 code sends the blocks into kernel pages where the
> kernel retrieves it and makes it available to system calls?
> 
> I am hoping that a detailed primer on how the ext2 works will give me
> some better background for studying the source code and understanding
> what's happening.  I'd like to get to a stage where I can modify the
> ext2fs code comfortably.

  By the  types of questions  you're asking, it  seems to me  that you
need to learn first, how filesystems interact with VFS, virtual memory
and the low-level block layer in Linux. After you have some feeling of
how that works (which is  common to most on-disk filesystems), you can
start worrying about ext2 specifics.

  The best documentation I've found on this is the book "Understanding
the Linux Kernel",  by D. P. Bovet & M. Cesati.  The newest edition is
the second, which  is suppose to cover kernel 2.4.  I've read over the
first edition, and it's really  good. Topics like VM, the block layer,
VFS and, in  particular, the ext2 file system, are  all covered in the
book. 

  Aside  from  that,  the kernelnewbies  site  (www.kernelnewbies.org),
specifically their "External Links" page
(www.kernelnewbies.org/links/)  has a  valuable list  of documentation
sites, some even regarding filesystems in general, and the ext2fs. 

> I've read the design and implementation docs on the e2fsprogs sf site,
> and the kernel source doc on ext2 which lists that as a reference.
> There is a reference to a site which has some code to compress data on
> ext2 filesystems, and that would be really ideal (see below) but
> apparently it's no longer maintained or supported.
> 
> What I want to do as a research project is implement compression.
> Essentially the code has to compress and decompress on the fly.  I
> would imagine compression would be limited to each block individually,
> otherwise data on other blocks would be needed to decompress a single
> block.  So I need to hook into the part of ext2 where the data flies
> through on its journey back and forth from the kernel to the physical
> disk.  I'd also like to implement secure deletion by wiping blocks
> when they're freed so recovery is impossible, and other experiments.

  Just as  a note,  there are some  other compressed file  systems for
Linux floating around.... For example:

E2compress project:
  http://www.alizt.com/index.html

SquashFS:
  http://squashfs.sourceforge.net/

CramFS:
  http://sourceforge.net/projects/cramfs/

  And perhaps others... Google around  a bit. The advantage of looking
at these  other projects, is  that you learn  a bit of what  they have
done, and why, etc.

> PS.  How do fs developers debug their code?  Do they use syslog, or
> can they somehow run the fs code in debugger like gdb?

  Personally, printing stuff to syslog  is not very efficient, in most
cases.  If you  really intend  to do  kernel hacking,  you  should get
acquainted with  some virtual machines.  The order of  (my) preference
is:

User mode Linux:
  http://user-mode-linux.sf.net/

Bochs Emulator:
  http://bochs.sf.net/

  There   are,   of  course,   some   kernel   debuggers,  like   kgdb
(kgdb.sourceforge.net/)  and  kdb  (oss.sgi.com/projects/kdb/), but  I
find that  they are only useful when  your work is mature  and you are
ready to do some testing on real machines.


  good luck,

--  
  Livio B. Soares