libaal documentation: fact or fiction?

libaal documentation: fact or fiction?

[Geoff Oakham]

Hello,

I'm interested in documenting libaal for reiserfs4.  Is anyone currently 
working on documenting the public API?  If not, I'd like to take a stab 
at it.

If someone could direct me to any existing documentation and/or test 
cases and/or sample code and/or images of napkins used by the author.. I 
would be most greatful.

Thanks in advance,

Geoff

Re: libaal documentation: fact or fiction?

[Yury Umanets]

Geoff Oakham wrote:

> Hello, 

Hello Geoff,

>
>
> I'm interested in documenting libaal for reiserfs4.  Is anyone 
> currently working on documenting the public API?  If not, I'd like to 
> take a stab at it.
>
> If someone could direct me to any existing documentation and/or test 
> cases and/or sample code and/or images of napkins used by the author.. 
> I would be most greatful.
>
> Thanks in advance,
>
> Geoff
>
>
>
>
>
First of all thanks for proposal.

Libaal has been developed as a try of standard interface between 
corresponding FS userspace library (currently libreiser4) and 
application that wants to use that library (currently reiser4progs).

Predecessor of libaal (mostly device abstraction) is used in libreiserfs 
(progsreiserfs package), parted, etc. And it showed, that it good to 
have an abstraction interface like it is.

And we have plans to use libaal for other filesystems too. For instance 
for ntfs (ntfstools project). So, we definitely  interested in 
documenting libaal. But unfortunatelly it is not documented yet. And if 
you want to help us, I don't see any barriers.

Take a look into soure first. Probably you'll propose something we have 
missed it, but useful for users. Ask questions, we will answer them.

BTW, why do you interested in documenting it?

-- 
umka

Re: libaal documentation: fact or fiction?

[Geoff Oakham]

Hi Yury,

Thanks for getting back to me!

On Wed, Oct 08, 2003 at 10:52:20AM +0400, Yury Umanets wrote:
> Libaal has been developed as a try of standard interface between 
> corresponding FS userspace library (currently libreiser4) and 
> application that wants to use that library (currently reiser4progs).

Is this library the userland interface applications use to write to
files in atomic operations?

> BTW, why do you interested in documenting it?

A number of reasons: I'm interested in ReiserFs4 and I want to start
playing with it.  As I understand it, Reiserfs4 will offer data
journalling and plugin support.. both things I'd like to play with.
Since neither seems to have been documented yet, I thought I'd start by
attempting that.

I'm also intersted because data-journalling has the potential to become
a standard interface for Linux filesystems.. and as an outsider I was
hoping I would be able to offer suggestions on how to clean up that API
and speed up its acceptance.  Futhermore, I want to see if it's possible
to implement the equivalent functionality (at least from a programmer's
perspective) on a traditional UNIX filesystem using old-fasioned
file-locking.

Cheers,

Geoff

-- 
  PGP fingerprint: 8ADC 92E1 6782 D034 E0E3  8EF4 EA4D 25E3 C17C 48D2
  "If you want to learn more about guns, get a job at [an American]
  convenience store or visit our website at ... " --Michael Moore

Re: libaal documentation: fact or fiction?

[Yury Umanets]

Geoff Oakham wrote:

>Hi Yury,
>
>Thanks for getting back to me!
>
>On Wed, Oct 08, 2003 at 10:52:20AM +0400, Yury Umanets wrote:
>  
>
>>Libaal has been developed as a try of standard interface between 
>>corresponding FS userspace library (currently libreiser4) and 
>>application that wants to use that library (currently reiser4progs).
>>    
>>
>
>Is this library the userland interface applications use to write to
>files in atomic operations?
>
No. It serves just for having persistent interface for working with 
device inside libreiser4.
Suppose we have to create some structures in /dev/hdb1 durring mkfs. And 
this is job for libreiser4. It calls fs_create() and it then calls 
object_create(fs, "/"), etc. And all these functions need to write and 
to read something from /dev/hdb1. But we can't use just file descriptor 
for writing and reading in libreiser4,  because it then will be bound to 
userspace and posix. What is one will want to use it for repair 
something in dos? Or GRUB will want to use libreiser4 (actually is does) 
to read kernel image from reiser4 durring boot? They will unable to do 
so. There also lots of another cases when raw file descriptor is not enough.

In order to solve all these issues, we have developed device abstraction 
mechanism. Which works like linux kernel drivers (roughly speaking).
That is we have instance of opened device with current device operations 
installed in it. And device have methods read() and write() etc. And 
when we call write(), device calls current wrote operation installed in 
it in the manner like the fiollowing:

int aal_device_write(aal_device_t *dev, char *buff, int off, int n) {
    if (!dev->ops->write)
       return -EINVAL;

    return dev->ops->write(dev->entity, buff, off, n);
}

Device initializing for working with file in usual enviromnent:

dev = aal_device_init("/dev/hdb1", &file_ops);


Device initializing in unusual enviromnent:
dev = aal_device_init("/dev/hdb1", &bios_irqs_using_ops);


and so on.

It is some kind of plugins, but they as not impelemented inside libaal 
(except of file_ops) and we expect, that user who need corresponding 
device abstraction (for instance for working on compressed filesystem 
image) will implement it himself.

>
>  
>
>>BTW, why do you interested in documenting it?
>>    
>>

>A number of reasons: I'm interested in ReiserFs4 and I want to start
>playing with it.  As I understand it, Reiserfs4 will offer data
>journalling and plugin support.. both things I'd like to play with.
>Since neither seems to have been documented yet, I thought I'd start by
>attempting that.
>
Ok, I see, but libreiser4 does not any things with journaling and so on. 
So, probably it is better to start from reading resier4 design documents 
on our website and then take a look to kernel code.


>
>I'm also intersted because data-journalling has the potential to become
>a standard interface for Linux filesystems.. and as an outsider I was
>hoping I would be able to offer suggestions on how to clean up that API
>and speed up its acceptance.
>
We will have userspace API for controlling transactions and other things 
latter when sys_reiser4() is finished.


>  Futhermore, I want to see if it's possible
>to implement the equivalent functionality (at least from a programmer's
>perspective) on a traditional UNIX filesystem using old-fasioned
>file-locking.
>
Can you say more detailed about this? What do you mean speaking about 
old-fasioned file-locking?

>
>Cheers,
>
>Geoff
>
>  
>
Thanks.

-- 
umka

Re: libaal documentation: fact or fiction?

[Geoff Oakham]

On Wed, Oct 08, 2003 at 10:12:27PM +0400, Yury Umanets wrote:
> >Is this library the userland interface applications use to write to
> >files in atomic operations?
> No. It serves just for having persistent interface for working with 
> device inside libreiser4.

So this is the library Lilo & friends would want to use to find out
where a kernel image is physically located on disk?  Thanks for setting
me straight on this one..  obviously I've guessed wrong.

> Ok, I see, but libreiser4 does not any things with journaling and so on. 
> So, probably it is better to start from reading resier4 design documents 
> on our website and then take a look to kernel code.

Although I find the design documents very interesting, I can't say I
find them helpful for navigating the source.  Perhaps I haven't read
them thoroughly enough.

> We will have userspace API for controlling transactions and other things 
> latter when sys_reiser4() is finished.

Ah, that's good to know.  What's the status of that work?

> > Futhermore, I want to see if it's possible
> >to implement the equivalent functionality (at least from a programmer's
> >perspective) on a traditional UNIX filesystem using old-fasioned
> >file-locking.
> >
> Can you say more detailed about this? What do you mean speaking about 
> old-fasioned file-locking?

Sure: Imagine I'm the author of 'mutt', a popular mail reader.  [I'm
not, but for this "use case," please imagine I am.]  'mutt' allows
multiple instances of itself to run in parallel.. which means I have to
be careful when reading and writing to the [poor] user's mailbox.
Traditionally, this meant I would have to group all my file access into
'critical sections' and obtain an appropriate file locks for the
duration of each section.

At the begining & end of the critical sections, I should be able to
assert the file is in a valid, "non-corrupt" state.  Wait... this sounds
suspiciously like a "transaction": if I wanted to modify mutt to take
advantage of reiserfs4's data journalling, the critical sections would
be the obvious place to do it.

Imagine that I've gone ahead and made these modifications.  My next
thought would be: "ok, now without using #ifdefs, how can I continue to
support legacy platforms [and/or filesystems] that don't support data
journalling?"  This is where I'd hope a good userspace library could
step in.

Does that make sense, or am I completely missing the point of data
journalling?

Thanks,

Geoff

-- 
  PGP fingerprint: 8ADC 92E1 6782 D034 E0E3  8EF4 EA4D 25E3 C17C 48D2
  "If you want to learn more about guns, get a job at [an American]
  convenience store or visit our website at ... " --Michael Moore

Re: libaal documentation: fact or fiction?

[Yury Umanets]

Geoff Oakham wrote:

>On Wed, Oct 08, 2003 at 10:12:27PM +0400, Yury Umanets wrote:
>  
>
>>>Is this library the userland interface applications use to write to
>>>files in atomic operations?
>>>      
>>>
>>No. It serves just for having persistent interface for working with 
>>device inside libreiser4.
>>    
>>
>
>So this is the library Lilo & friends would want to use to find out
>where a kernel image is physically located on disk?  Thanks for setting
>me straight on this one..  obviously I've guessed wrong.
>
Actually libaal is base library for libreiser4. It contains all needed 
type definitions, utils, device abstraction, and device data block 
functions, exceptions, etc. And libreiser4 along with it is supposed to 
provide reiser4 access in user space for any application, which would 
want to acess reiser4 without kernel. For instance, some repair tool 
like testdisk.

libreiser4 is able to create files, write and then delete them. The same 
about directories. It also has plugin based design and may be easyly 
expanded. For now, it contains all the code needed for working with 
reiser4 in userspace (or in so called stand alone mode, like GRUB works 
in it durring showing menu and reading kernel image).

We have a pach for GRUB, so it uses libreiser4 for provide reiser4 support.

>
>  
>
>>Ok, I see, but libreiser4 does not any things with journaling and so on. 
>>So, probably it is better to start from reading resier4 design documents 
>>on our website and then take a look to kernel code.
>>    
>>

>Although I find the design documents very interesting, I can't say I
>find them helpful for navigating the source.  Perhaps I haven't read
>them thoroughly enough.
>
That is because they descibe only main ideas, not the how these ideas 
are impelemnted in particular language or enviromnent. Be sure, if you 
ask me or someone else of our team, about reiser4 sources, we will 
answer any question.

>
>  
>
>>We will have userspace API for controlling transactions and other things 
>>latter when sys_reiser4() is finished.
>>    
>>
>
>Ah, that's good to know.  What's the status of that work?
>
Actually I'm not sure, that my information about the status is up 
to-date. Ask Hans about this.

>
>  
>
>>>Futhermore, I want to see if it's possible
>>>to implement the equivalent functionality (at least from a programmer's
>>>perspective) on a traditional UNIX filesystem using old-fasioned
>>>file-locking.
>>>
>>>      
>>>
>>Can you say more detailed about this? What do you mean speaking about 
>>old-fasioned file-locking?
>>    
>>
>
>Sure: Imagine I'm the author of 'mutt', a popular mail reader.  [I'm
>not, but for this "use case," please imagine I am.] 
>
Ok :)

> 'mutt' allows
>multiple instances of itself to run in parallel.. which means I have to
>be careful when reading and writing to the [poor] user's mailbox.
>

>Traditionally, this meant I would have to group all my file access into
>'critical sections' and obtain an appropriate file locks for the
>duration of each section.
>

>
>At the begining & end of the critical sections, I should be able to
>assert the file is in a valid, "non-corrupt" state.  Wait... this sounds
>suspiciously like a "transaction": if I wanted to modify mutt to take
>advantage of reiserfs4's data journalling, the sections would
>be the obvious place to do it.
>
Hm.. It's cool idea. There only a thing... It is needed to rewrite mutt 
and other tools to use this critical section. Also,  this should be 
system wide, I mean, that linux should support this feature in its VFS. 
I guess Hans will be interesting to discuss this more detailed.

Yet another method that comed to my mind, is to impelement ability to 
open not the whole file, but some its part and to have separated file 
descriptor for that file. It seems, that QNX is able to do so.

>
>Imagine that I've gone ahead and made these modifications.  My next
>thought would be: "ok, now without using #ifdefs, how can I continue to
>support legacy platforms [and/or filesystems] that don't support data
>journalling?"  This is where I'd hope a good userspace library could
>step in.
>
So, you propose to use some kind of crossover between applications and 
actual call to kernel? And this bridge should decide is backend 
filesystem is able to do journaling and has user space journaling 
control API? It is yet another good idea.


>
>Does that make sense, or am I completely missing the point of data
>journalling?
>
This makes sence and all the things you've described are the use of 
journaling control in user space.

>
>Thanks,
>
>Geoff
>
>  
>


-- 
umka

Re: libaal documentation: fact or fiction?

[Geoff Oakham]

On Wed, Oct 08, 2003 at 11:56:55PM +0400, Yury Umanets wrote:
> >>We will have userspace API for controlling transactions and other things 
> >>latter when sys_reiser4() is finished.
> >Ah, that's good to know.  What's the status of that work?
> Actually I'm not sure, that my information about the status is up 
> to-date. Ask Hans about this.

Does Hans monitor all traffic here, or should I explicitly ping() him?

> Hm.. It's cool idea. There only a thing... It is needed to rewrite mutt 
> and other tools to use this critical section. Also,  this should be 
> system wide, I mean, that linux should support this feature in its VFS. 
> I guess Hans will be interesting to discuss this more detailed.

*nod* I think Hans mentioned something about trying out potential
changes to VFS initially on reiserfs.. somewhere in the documentation.
(I can't remember where.)

> Yet another method that comed to my mind, is to impelement ability to 
> open not the whole file, but some its part and to have separated file 
> descriptor for that file. It seems, that QNX is able to do so.

That sounds oddly familiar.  Didn't one of the dozens of attempts to
standardize file locking protocols consider this approch?  (I know enough
about file locking to know I should avoid it.)

I seem to remember NFS to be a particularily dodgy spot for this stuff.

> So, you propose to use some kind of crossover between applications and 
> actual call to kernel? And this bridge should decide is backend 
> filesystem is able to do journaling and has user space journaling 
> control API? It is yet another good idea.

Umm.. yes?  (I'm sorry, I'm not sure I follow, so I'll ramble a bit and
hope I mention something approaching an answer.)

I'd like this library to behave properly in most situations:

  a. Reiser supports transactions, kernel does not [yet?].
  b. VFS supports transactions
  c. OS is not Linux and/or does not support transactions at all.

  i. filesystem supports transactions
  ii. filesystem does not support transactions

Where exactly you put the code that chooses the appropriate
implementation doesn't matter.  I think it will vary depending on the
platform & situation.. regardless, it should be hidden from the user.
Obviously in situation #c, it would have to be handled at compile time.
In #b, the kernel would handle it.  #a.. perhaps both user and kernel
will be needed..?

Geoff

-- 
  PGP fingerprint: 8ADC 92E1 6782 D034 E0E3  8EF4 EA4D 25E3 C17C 48D2
  "If you want to learn more about guns, get a job at [an American]
  convenience store or visit our website at ... " --Michael Moore

Re: libaal documentation: fact or fiction?

[Yury Umanets]

Geoff Oakham wrote:

>On Wed, Oct 08, 2003 at 11:56:55PM +0400, Yury Umanets wrote:
>  
>
>>>>We will have userspace API for controlling transactions and other things 
>>>>latter when sys_reiser4() is finished.
>>>>        
>>>>
>>>Ah, that's good to know.  What's the status of that work?
>>>      
>>>
>>Actually I'm not sure, that my information about the status is up 
>>to-date. Ask Hans about this.
>>    
>>
>
>Does Hans monitor all traffic here, or should I explicitly ping() him?
>
Ping him explicitly please :)

>
>  
>
>>Hm.. It's cool idea. There only a thing... It is needed to rewrite mutt 
>>and other tools to use this critical section. Also,  this should be 
>>system wide, I mean, that linux should support this feature in its VFS. 
>>I guess Hans will be interesting to discuss this more detailed.
>>    
>>
>
>*nod* I think Hans mentioned something about trying out potential
>changes to VFS initially on reiserfs.. somewhere in the documentation.
>(I can't remember where.)
>

>
>  
>
>>Yet another method that comed to my mind, is to impelement ability to 
>>open not the whole file, but some its part and to have separated file 
>>descriptor for that file. It seems, that QNX is able to do so.
>>    
>>

>That sounds oddly familiar.  Didn't one of the dozens of attempts to
>standardize file locking protocols consider this approch?  (I know enough
>about file locking to know I should avoid it.)
>
>I seem to remember NFS to be a particularily dodgy spot for this stuff.
>
>  
>
>>So, you propose to use some kind of crossover between applications and 
>>actual call to kernel? And this bridge should decide is backend 
>>filesystem is able to do journaling and has user space journaling 
>>control API? It is yet another good idea.
>>    
>>
>
>Umm.. yes?  (I'm sorry, I'm not sure I follow, so I'll ramble a bit and
>hope I mention something approaching an answer.)
>
>I'd like this library to behave properly in most situations:
>
>  a. Reiser supports transactions, kernel does not [yet?].
>
Reiserfs is part of kernel acually. It is running on the same privilage 
level.

>  b. VFS supports transactions
>  c. OS is not Linux and/or does not support transactions at all.
>
>  i. filesystem supports transactions
>  ii. filesystem does not support transactions
>  
>

>Where exactly you put the code that chooses the appropriate
>implementation doesn't matter.  I think it will vary depending on the
>platform & situation.. regardless, it should be hidden from the user.
>Obviously in situation #c, it would have to be handled at compile time.
>In #b, the kernel would handle it.  #a.. perhaps both user and kernel
>will be needed..?
>

Transactions are exist in reiserfs and reiser4 and xfs, etc. But they 
are not controlled from user space. You can't say roolback() after some 
fs actions. And we are speaking about the way how to control them. 
Probably using system call as a interface is good idea.

Speaking about the library... It is fully userspace solution. It might 
be like libc.

>
>Geoff
>
>  
>


-- 
umka

Re: libaal documentation: fact or fiction?

[Hans Reiser]

Geoff Oakham wrote:

>Hi Yury,
>
>Thanks for getting back to me!
>
>On Wed, Oct 08, 2003 at 10:52:20AM +0400, Yury Umanets wrote:
>  
>
>>Libaal has been developed as a try of standard interface between 
>>corresponding FS userspace library (currently libreiser4) and 
>>application that wants to use that library (currently reiser4progs).
>>    
>>
>
>Is this library the userland interface applications use to write to
>files in atomic operations?
>
No, sys_reiser4 is.  sys_reiser4 is currently in a quite raw state.  You 
might want to contact us again in 3 months.

We do not support isolation in transactions yet.  We only support 
atomicity.  Atomicity is easier to implement without complications 
regarding performance and deadlock than isolation.

>
>  
>
>>BTW, why do you interested in documenting it?
>>    
>>
>
>A number of reasons: I'm interested in ReiserFs4 and I want to start
>playing with it.  As I understand it, Reiserfs4 will offer data
>journalling and plugin support.. both things I'd like to play with.
>Since neither seems to have been documented yet, I thought I'd start by
>attempting that.
>
>I'm also intersted because data-journalling has the potential to become
>a standard interface for Linux filesystems.. and as an outsider I was
>hoping I would be able to offer suggestions on how to clean up that API
>and speed up its acceptance.  Futhermore, I want to see if it's possible
>to implement the equivalent functionality (at least from a programmer's
>perspective) on a traditional UNIX filesystem using old-fasioned
>file-locking.
>
>Cheers,
>
>Geoff
>
>  
>


-- 
Hans

Re: libaal documentation: fact or fiction?

[Geoff Oakham]

On Thu, Oct 09, 2003 at 02:19:02PM +0400, Hans Reiser wrote:
> >Is this library the userland interface applications use to write to
> >files in atomic operations?
> No, sys_reiser4 is.  sys_reiser4 is currently in a quite raw state.  You 
> might want to contact us again in 3 months.

"Raw" doesn't bother me.. in fact I'd be interested in being a Guinea
pig.  Is it in BK, or is there a snapshot I can grab?

Thanks,

Geoff

-- 
  PGP fingerprint: 8ADC 92E1 6782 D034 E0E3  8EF4 EA4D 25E3 C17C 48D2
  "If you want to learn more about guns, get a job at [an American]
  convenience store or visit our website at ... " --Michael Moore