Re: Comments about indexing and query solving in user land

Re: Comments about indexing and query solving in user land

[Grzegorz Jaśkiewicz]

On Mon, 06 Sep 2004 14:32:34 +0530, Faraz S <farazs@calsoftinc.com> wrote:
>         In our project http://sourceforge.net/projects/linuxsfs-tm/ we did the
> attribute indexing and query parsing in the user-space and we had hooks
> in the VFS which communicated with the user-space deamons. As far as
> directory listing is concerned we had a in kernel-recordset cache
> (hashed on the query). The listing were pretty efficient.
> 
>         But the real problem occured when a large number of files were copied
> in the onto our file system. All the files now need to be indexed in the
> user space , leading to mode switch. That was pathetically slooow. The
> entire 2.4 sources were copied in 5-6 hours. I think this problem could
> be solved by batching requests the user space deamon.  Then u have all
> the problems on missing out on indexing on a large set of files.
I think that should give you pretty much explanation why whole VFS
layer and filesystems are in kernel not in user space. That's where
such sort of thing belongs to. I have no doubts.
  


-- 
GJ

Re: Comments about indexing and query solving in user land

[daniel.poelzleithner]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Grzegorz Jaśkiewicz wrote:

|>        In our project http://sourceforge.net/projects/linuxsfs-tm/ we
did the
|>attribute indexing and query parsing in the user-space and we had hooks
|>in the VFS which communicated with the user-space deamons. As far as
|>directory listing is concerned we had a in kernel-recordset cache
|>(hashed on the query). The listing were pretty efficient.
|>
|>        But the real problem occured when a large number of files were
copied
|>in the onto our file system. All the files now need to be indexed in the
|>user space , leading to mode switch. That was pathetically slooow. The
|>entire 2.4 sources were copied in 5-6 hours. I think this problem could
|>be solved by batching requests the user space deamon.  Then u have all
|>the problems on missing out on indexing on a large set of files.
|
| I think that should give you pretty much explanation why whole VFS
| layer and filesystems are in kernel not in user space. That's where
| such sort of thing belongs to. I have no doubts.

I think such operations should be a background task. When a new file is
created, the file should be added to a index queue and the file
operation should return. These queue list will be processed by an
background daemon.
Only when meta data are accessed which is unindexed the file will be
removed from the queue and proccessed instandly, so the system call can
return.
I think such a technique would reduce the speed impact indexing brings.

regards
~ Daniel

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBPhV1y/mkIQp7AD0RAqonAJ9ww2VAPKnX7u9q1euyKZbYBK9zGQCg2fI9
mC6bQ5IiRuafCrX+O3I7oJw=
=Q8mj
-----END PGP SIGNATURE-----

Comments about indexing and query solving in user land

[Faraz S]

Hi;
	With the hot debate on the linux kernel mailing list about what
features must be in the kernel, i'd like to add my experience about
keeping the query parser in the userspace. 

	In our project http://sourceforge.net/projects/linuxsfs-tm/ we did the
attribute indexing and query parsing in the user-space and we had hooks
in the VFS which communicated with the user-space deamons. As far as
directory listing is concerned we had a in kernel-recordset cache
(hashed on the query). The listing were pretty efficient.

	But the real problem occured when a large number of files were copied
in the onto our file system. All the files now need to be indexed in the
user space , leading to mode switch. That was pathetically slooow. The
entire 2.4 sources were copied in 5-6 hours. I think this problem could
be solved by batching requests the user space deamon.  Then u have all
the problems on missing out on indexing on a large set of files.
	
	No doubt the feature which did content based listing was very well
recieved by the audience of the project. I will soon be uplaoding the
project documentation and PPT's. 

WE NEED THIS FOR SURE - BUT DONE IN THE RIGHT WAY

regards.

-- 
Faraz Ahmed Shaikh
Member Technical Staff
Calsoft Pvt Limited