Ext2 detailed documentation

Ext2 detailed documentation

[James Buchanan]

Aside from reading the source code (and fully understanding it, which
I am not sure I am capable of yet), is there any detailed
documentation on how the Ext2 fs works?  Specifically, I'd like to
know some fairly high-level details so I know what to look for when
I'm studying the sources.

Here's what I'm looking for:  where's the code that gets blocks of
data off the disk and sends it up through the kernel via the Ext2 fs
code?  And vice versa.  I see a lot of code dealing with pages, does
this mean the ext2 code sends the blocks into kernel pages where the
kernel retrieves it and makes it available to system calls?

I am hoping that a detailed primer on how the ext2 works will give me
some better background for studying the source code and understanding
what's happening.  I'd like to get to a stage where I can modify the
ext2fs code comfortably.

I've read the design and implementation docs on the e2fsprogs sf site,
and the kernel source doc on ext2 which lists that as a reference.
There is a reference to a site which has some code to compress data on
ext2 filesystems, and that would be really ideal (see below) but
apparently it's no longer maintained or supported.

What I want to do as a research project is implement compression.
Essentially the code has to compress and decompress on the fly.  I
would imagine compression would be limited to each block individually,
otherwise data on other blocks would be needed to decompress a single
block.  So I need to hook into the part of ext2 where the data flies
through on its journey back and forth from the kernel to the physical
disk.  I'd also like to implement secure deletion by wiping blocks
when they're freed so recovery is impossible, and other experiments.

PS.  How do fs developers debug their code?  Do they use syslog, or
can they somehow run the fs code in debugger like gdb?

Many thanks :)
James

Re: Ext2 detailed documentation

[Livio Baldini Soares]

  Hey James,

James Buchanan writes:
> Aside from reading the source code (and fully understanding it, which
> I am not sure I am capable of yet), is there any detailed
> documentation on how the Ext2 fs works?  Specifically, I'd like to
> know some fairly high-level details so I know what to look for when
> I'm studying the sources.
> 
> Here's what I'm looking for:  where's the code that gets blocks of
> data off the disk and sends it up through the kernel via the Ext2 fs
> code?  And vice versa.  I see a lot of code dealing with pages, does
> this mean the ext2 code sends the blocks into kernel pages where the
> kernel retrieves it and makes it available to system calls?
> 
> I am hoping that a detailed primer on how the ext2 works will give me
> some better background for studying the source code and understanding
> what's happening.  I'd like to get to a stage where I can modify the
> ext2fs code comfortably.

  By the  types of questions  you're asking, it  seems to me  that you
need to learn first, how filesystems interact with VFS, virtual memory
and the low-level block layer in Linux. After you have some feeling of
how that works (which is  common to most on-disk filesystems), you can
start worrying about ext2 specifics.

  The best documentation I've found on this is the book "Understanding
the Linux Kernel",  by D. P. Bovet & M. Cesati.  The newest edition is
the second, which  is suppose to cover kernel 2.4.  I've read over the
first edition, and it's really  good. Topics like VM, the block layer,
VFS and, in  particular, the ext2 file system, are  all covered in the
book. 

  Aside  from  that,  the kernelnewbies  site  (www.kernelnewbies.org),
specifically their "External Links" page
(www.kernelnewbies.org/links/)  has a  valuable list  of documentation
sites, some even regarding filesystems in general, and the ext2fs. 

> I've read the design and implementation docs on the e2fsprogs sf site,
> and the kernel source doc on ext2 which lists that as a reference.
> There is a reference to a site which has some code to compress data on
> ext2 filesystems, and that would be really ideal (see below) but
> apparently it's no longer maintained or supported.
> 
> What I want to do as a research project is implement compression.
> Essentially the code has to compress and decompress on the fly.  I
> would imagine compression would be limited to each block individually,
> otherwise data on other blocks would be needed to decompress a single
> block.  So I need to hook into the part of ext2 where the data flies
> through on its journey back and forth from the kernel to the physical
> disk.  I'd also like to implement secure deletion by wiping blocks
> when they're freed so recovery is impossible, and other experiments.

  Just as  a note,  there are some  other compressed file  systems for
Linux floating around.... For example:

E2compress project:
  http://www.alizt.com/index.html

SquashFS:
  http://squashfs.sourceforge.net/

CramFS:
  http://sourceforge.net/projects/cramfs/

  And perhaps others... Google around  a bit. The advantage of looking
at these  other projects, is  that you learn  a bit of what  they have
done, and why, etc.

> PS.  How do fs developers debug their code?  Do they use syslog, or
> can they somehow run the fs code in debugger like gdb?

  Personally, printing stuff to syslog  is not very efficient, in most
cases.  If you  really intend  to do  kernel hacking,  you  should get
acquainted with  some virtual machines.  The order of  (my) preference
is:

User mode Linux:
  http://user-mode-linux.sf.net/

Bochs Emulator:
  http://bochs.sf.net/

  There   are,   of  course,   some   kernel   debuggers,  like   kgdb
(kgdb.sourceforge.net/)  and  kdb  (oss.sgi.com/projects/kdb/), but  I
find that  they are only useful when  your work is mature  and you are
ready to do some testing on real machines.


  good luck,

--  
  Livio B. Soares

Re: Ext2 detailed documentation

[Mohammad Nayyer Zubair]

hii.

There is a brief but excellent description of the VFS at
http://www1.cs.columbia.edu/~ezk/research/software/vfs.txt

Check out zlib compression/decompression library at www.zlib.org.

And have you heard about Vmware?  Essentially its like an emulator. You
can run an entire OS inside this software, and hence fully test out your
file system. You can download a trial version from www.vmware.com .



On Fri, 4 Jul 2003, Livio Baldini Soares wrote:


>   Hey James,
>
> James Buchanan writes:
> > Aside from reading the source code (and fully understanding it, which
> > I am not sure I am capable of yet), is there any detailed
> > documentation on how the Ext2 fs works?  Specifically, I'd like to
> > know some fairly high-level details so I know what to look for when
> > I'm studying the sources.
> >
> > Here's what I'm looking for:  where's the code that gets blocks of
> > data off the disk and sends it up through the kernel via the Ext2 fs
> > code?  And vice versa.  I see a lot of code dealing with pages, does
> > this mean the ext2 code sends the blocks into kernel pages where the
> > kernel retrieves it and makes it available to system calls?
> >
> > I am hoping that a detailed primer on how the ext2 works will give me
> > some better background for studying the source code and understanding
> > what's happening.  I'd like to get to a stage where I can modify the
> > ext2fs code comfortably.
>
>   By the  types of questions  you're asking, it  seems to me  that you
> need to learn first, how filesystems interact with VFS, virtual memory
> and the low-level block layer in Linux. After you have some feeling of
> how that works (which is  common to most on-disk filesystems), you can
> start worrying about ext2 specifics.
>
>   The best documentation I've found on this is the book "Understanding
> the Linux Kernel",  by D. P. Bovet & M. Cesati.  The newest edition is
> the second, which  is suppose to cover kernel 2.4.  I've read over the
> first edition, and it's really  good. Topics like VM, the block layer,
> VFS and, in  particular, the ext2 file system, are  all covered in the
> book.
>
>   Aside  from  that,  the kernelnewbies  site  (www.kernelnewbies.org),
> specifically their "External Links" page
> (www.kernelnewbies.org/links/)  has a  valuable list  of documentation
> sites, some even regarding filesystems in general, and the ext2fs.
>
> > I've read the design and implementation docs on the e2fsprogs sf site,
> > and the kernel source doc on ext2 which lists that as a reference.
> > There is a reference to a site which has some code to compress data on
> > ext2 filesystems, and that would be really ideal (see below) but
> > apparently it's no longer maintained or supported.
> >
> > What I want to do as a research project is implement compression.
> > Essentially the code has to compress and decompress on the fly.  I
> > would imagine compression would be limited to each block individually,
> > otherwise data on other blocks would be needed to decompress a single
> > block.  So I need to hook into the part of ext2 where the data flies
> > through on its journey back and forth from the kernel to the physical
> > disk.  I'd also like to implement secure deletion by wiping blocks
> > when they're freed so recovery is impossible, and other experiments.
>
>   Just as  a note,  there are some  other compressed file  systems for
> Linux floating around.... For example:
>
> E2compress project:
>   http://www.alizt.com/index.html
>
> SquashFS:
>   http://squashfs.sourceforge.net/
>
> CramFS:
>   http://sourceforge.net/projects/cramfs/
>
>   And perhaps others... Google around  a bit. The advantage of looking
> at these  other projects, is  that you learn  a bit of what  they have
> done, and why, etc.
>
> > PS.  How do fs developers debug their code?  Do they use syslog, or
> > can they somehow run the fs code in debugger like gdb?
>
>   Personally, printing stuff to syslog  is not very efficient, in most
> cases.  If you  really intend  to do  kernel hacking,  you  should get
> acquainted with  some virtual machines.  The order of  (my) preference
> is:
>
> User mode Linux:
>   http://user-mode-linux.sf.net/
>
> Bochs Emulator:
>   http://bochs.sf.net/
>
>   There   are,   of  course,   some   kernel   debuggers,  like   kgdb
> (kgdb.sourceforge.net/)  and  kdb  (oss.sgi.com/projects/kdb/), but  I
> find that  they are only useful when  your work is mature  and you are
> ready to do some testing on real machines.
>
>
>   good luck,
>
> --
>   Livio B. Soares
> -
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Ext2 detailed documentation

[Randy.Dunlap]

On Sat, 5 Jul 2003 01:54:15 -0400 (EDT) Mohammad Nayyer Zubair <mzubair@ic.sunysb.edu> wrote:

| There is a brief but excellent description of the VFS at
| http://www1.cs.columbia.edu/~ezk/research/software/vfs.txt
| 
| Check out zlib compression/decompression library at www.zlib.org.

...

| > E2compress project:
| >   http://www.alizt.com/index.html
| >
| > SquashFS:
| >   http://squashfs.sourceforge.net/
| >
| > CramFS:
| >   http://sourceforge.net/projects/cramfs/


Hi,
Thanks for the links.

With linux-fsdevel lacking any home page collection AFAIK,
I've assembled some info (links) at
  http://www.xenotime.net/linux/linux-fs.html
However, I haven't updated this in a while... I need to add these
and some others.

--
~Randy
| http://developer.osdl.org/rddunlap/ | http://www.xenotime.net/linux/ |

Re: Ext2 detailed documentation

[Herbert Poetzl]

On Sun, Jul 06, 2003 at 11:42:20AM -0700, Randy.Dunlap wrote:
> On Sat, 5 Jul 2003 01:54:15 -0400 (EDT) Mohammad Nayyer Zubair <mzubair@ic.sunysb.edu> wrote:
> 
> | There is a brief but excellent description of the VFS at
> | http://www1.cs.columbia.edu/~ezk/research/software/vfs.txt
> | 
> | Check out zlib compression/decompression library at www.zlib.org.
> 
> ...
> 
> | > E2compress project:
> | >   http://www.alizt.com/index.html
> | >
> | > SquashFS:
> | >   http://squashfs.sourceforge.net/
> | >
> | > CramFS:
> | >   http://sourceforge.net/projects/cramfs/
> 
> 
> Hi,
> Thanks for the links.
> 
> With linux-fsdevel lacking any home page collection AFAIK,
> I've assembled some info (links) at
>   http://www.xenotime.net/linux/linux-fs.html
> However, I haven't updated this in a while... I need to add these
> and some others.

nice collection anyway ...

thanks,
Herbert

> --
> ~Randy
> | http://developer.osdl.org/rddunlap/ | http://www.xenotime.net/linux/ |
> -
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html