repost for readdir

repost for readdir

[jenn sirp]

Hi,

I'd like to thank you guys for your help earlier this year with the
initial phase of my project. I was able to implement FIFO queueing
functionality in Reiser v3. It is not very elegant, but I am still
learning. I was able to steer my supervisor in the direction of Reiser4.
It appears that the design principles of the new system will make it very
valuable for many at the Lab.

That said, and back to Hans' suggestion of a readdir implementation for
FIFO and LIFO file queuing... I have some questions. If anyone can answer
them, has advice, or suggestions that would be so great.

Here's where I am at, and what I don't know:

From what I can tell, files in the reiserfs 'Directory Item' aren't preserved in
the order they are written due to deletes and other operations. So if I
were to access the Directory Item to generate a list for readdir I couldn't be certain that they would be in first
created -to- last created order.

If readdir has to sort the files in the directory every time it is called
it seems like it would be a bit of a performance hit. Should another
structure be maintained that preserves the order as the files are
written... and should readdir use that as it's basis for returning the
directory contents?

If this is a question that doesn't belong here... please let me know and I apologize.

Thanks,

Jenn

Re: repost for readdir

[Vladimir V. Saveliev]

Hello

jenn sirp wrote:
> Hi,
> 
> I'd like to thank you guys for your help earlier this year with the
> initial phase of my project. I was able to implement FIFO queueing
> functionality in Reiser v3. It is not very elegant, but I am still
> learning. I was able to steer my supervisor in the direction of Reiser4.
> It appears that the design principles of the new system will make it very
> valuable for many at the Lab.
> 
> That said, and back to Hans' suggestion of a readdir implementation for
> FIFO and LIFO file queuing... I have some questions. If anyone can answer
> them, has advice, or suggestions that would be so great.
> 
> Here's where I am at, and what I don't know:
> 
>>From what I can tell, files in the reiserfs 'Directory Item' aren't preserved in
> the order they are written due to deletes and other operations. So if I
> were to access the Directory Item to generate a list for readdir I couldn't be certain that they would be in first
> created -to- last created order.
> 
> If readdir has to sort the files in the directory every time it is called
> it seems like it would be a bit of a performance hit.

Do you mean readdir library function? It reads directory and fills 
dirent structures. It does not sort anything. It even does not stat 
files names of which it reads.

  Should another
> structure be maintained that preserves the order as the files are
> written... and should readdir use that as it's basis for returning the
> directory contents?
> 

Are you trying to optimize ls -l?

Please describe in more details what are you trying to do.

> If this is a question that doesn't belong here... please let me know and I apologize.
> 
> Thanks,
> 
> Jenn
> 
> 
> 

Re: repost for readdir

[Hans Reiser]

Vladimir V. Saveliev wrote:

> Hello
>
> jenn sirp wrote:
>
>> Hi,
>>
>> I'd like to thank you guys for your help earlier this year with the
>> initial phase of my project. I was able to implement FIFO queueing
>> functionality in Reiser v3. It is not very elegant, but I am still
>> learning. I was able to steer my supervisor in the direction of Reiser4.
>> It appears that the design principles of the new system will make it 
>> very
>> valuable for many at the Lab.
>>
>> That said, and back to Hans' suggestion of a readdir implementation for
>> FIFO and LIFO file queuing... I have some questions. If anyone can 
>> answer
>> them, has advice, or suggestions that would be so great.
>>
>> Here's where I am at, and what I don't know:
>>
>>> From what I can tell, files in the reiserfs 'Directory Item' aren't 
>>> preserved in
>>
>> the order they are written due to deletes and other operations. So if I
>> were to access the Directory Item to generate a list for readdir I 
>> couldn't be certain that they would be in first
>> created -to- last created order.
>
We sort in hash order, not creation order.

>>
>> If readdir has to sort the files in the directory every time it is 
>> called
>> it seems like it would be a bit of a performance hit.
>
>
> Do you mean readdir library function? It reads directory and fills 
> dirent structures. It does not sort anything. It even does not stat 
> files names of which it reads.
>
> Should another
>
>> structure be maintained that preserves the order as the files are
>> written... and should readdir use that as it's basis for returning the
>> directory contents?
>>
>
> Are you trying to optimize ls -l?
>
> Please describe in more details what are you trying to do.
>
>> If this is a question that doesn't belong here... please let me know 
>> and I apologize.
>>
>> Thanks,
>>
>> Jenn
>>
>>
>>
>
>
>
>
Please consider working with V4, it will be easier and more useful I think.

Re: repost for readdir

[jenn sirp]

> 
> Do you mean readdir library function? It reads directory and fills 
> dirent structures. It does not sort anything. It even does not stat 
> files names of which it reads.

My project here was to alter reiserFS for automatic file queueing. When
files are written their names are automatically placed in a linked list
-- or queue in FIFO or LIFO order. A call to ioctl will return the next
file from the queue. This works for directories too.(every dir has it's
own set of queues). These queues, or lists, reflect the state that the
files are in i.e. enqueued, processing, error, etc.. Every time a file
is written in with a certain prefix the queues for that directory are
updated accordingly.

The purpose behind this type of system is to speed up file processing on
a server that receives queries and sends requests onto a database to be
processed.

I thought that Hans had mentioned that he would accept a readdir plug-in
for Reiser4. I took this to mean the reiserfs_readdir function. But
perhaps I misunderstood. 

Thanks so much,

Jenn

Re: repost for readdir

[jenn sirp]

> Please consider working with V4, it will be easier and more useful I
think.

I plan to. I read the documentation on-line about the plug-in capability
of your FS -- very cool. And I have the rest of the summer to poke
around with this idea. 

My questions are related to the implementation from a performance
perspective... If readdir is the only function to be altered... it seems
there would be a bit of a performance hit because a linear search, or
binary at least would need to be performed every time. 

I am also not certain as to how to determine 'when' the file was created
I assume it would need to be determined from the inode creation time?

I am just looking for some general pointers as to implementation of this
would best be achieved. I thought that maintaining another structure
somewhere else in the code, that preserved filenames as they were
written into the FS, would speed up readdir. 

Hopefully some of that makes sense ; )

Jenn