15M files

15M files

[studdugie]

Hello. I'm looking to replace a couple Berkeley DB data stores w/
regular file system directories backed by reiserfs (3.6). The reason
is Berkeley DB is slow especially for data w/ little or no locality of
reference. I'm posting to this list because I would like to get some
opinions on if reiserfs is suitable for the job. Currently there are
15,079,597 records in 1 of the database. If I moved to a directory
based db it would result in 15,079,597 discreet files ranging in sizes
from 1 byte to 1kb. I was reading the FAQ on the namesys site and it
mentioned that the r5 hash supports 1,200,000 files w/o collision.
Since 15M is 12.5x greater I'm expecting massive amounts of
collisions. So the question becomes how bad should I expect it to be?
Should I assume the file system can handle it or slow to a crawl?  I
would really appreciate some feedback from the experts before I go
ripping out the Berkeley DB code.

Thanx.

Re: 15M files

[PFC]

On Reiser4, I eperienced a really massive speedup when switching from  
berkeley to plain filesystem on my subversion repository.
But this is on reiser4. Try it ;)

Also you can really easily defragment your reiserfs database-directory :  
just tar and untar.


On Fri, 19 Aug 2005 23:49:40 +0200, studdugie <studdugie@gmail.com> wrote:

> Hello. I'm looking to replace a couple Berkeley DB data stores w/
> regular file system directories backed by reiserfs (3.6). The reason
> is Berkeley DB is slow especially for data w/ little or no locality of
> reference. I'm posting to this list because I would like to get some
> opinions on if reiserfs is suitable for the job. Currently there are
> 15,079,597 records in 1 of the database. If I moved to a directory
> based db it would result in 15,079,597 discreet files ranging in sizes
> from 1 byte to 1kb. I was reading the FAQ on the namesys site and it
> mentioned that the r5 hash supports 1,200,000 files w/o collision.
> Since 15M is 12.5x greater I'm expecting massive amounts of
> collisions. So the question becomes how bad should I expect it to be?
> Should I assume the file system can handle it or slow to a crawl?  I
> would really appreciate some feedback from the experts before I go
> ripping out the Berkeley DB code.
>
> Thanx.
>

Re: 15M files

[David Masover]

studdugie wrote:
> Hello. I'm looking to replace a couple Berkeley DB data stores w/
> regular file system directories backed by reiserfs (3.6).

Why 3.6, and not 4?  I'll bet v4 is better at this.

> I
> would really appreciate some feedback from the experts before I go
> ripping out the Berkeley DB code.

Others can answer the questions about whether the FS can handle it.  I
would suggest that you consider a real database, though, at least until
Reiser4 gets a good way (sys_reiser4?) of handling multiple files --
otherwise, you can expect to lose a lot of speed to all the open(2) calls.

I don't know anything about the status of sys_reiser4.  If it's almost
done, you may want to wait for it, but I don't think it's almost done.

Maybe MySQL would do the trick?  You may end up having to benchmark it
yourself...

Re: 15M files

[studdugie]

MySQL is too slow for the task, even over localhost. I've benchmarked
it. Berkeley DB was 3x faster than MySQL for my usecase. The app is
written in Java so I would be forced to use JDBC to talk to MySQL. The
wrapping/unwrapping overhead is a killer.

On 8/19/05, David Masover <ninja@slaphack.com> wrote:
> 
> studdugie wrote:
> > Hello. I'm looking to replace a couple Berkeley DB data stores w/
> > regular file system directories backed by reiserfs (3.6).
> 
> Why 3.6, and not 4?  I'll bet v4 is better at this.
> 
> > I
> > would really appreciate some feedback from the experts before I go
> > ripping out the Berkeley DB code.
> 
> Others can answer the questions about whether the FS can handle it.  I
> would suggest that you consider a real database, though, at least until
> Reiser4 gets a good way (sys_reiser4?) of handling multiple files --
> otherwise, you can expect to lose a lot of speed to all the open(2) calls.
> 
> I don't know anything about the status of sys_reiser4.  If it's almost
> done, you may want to wait for it, but I don't think it's almost done.
> 
> Maybe MySQL would do the trick?  You may end up having to benchmark it
> yourself...
>

Re: 15M files

[Christian Iversen]

On Saturday 20 August 2005 00:44, studdugie wrote:
> MySQL is too slow for the task, even over localhost. I've benchmarked
> it. Berkeley DB was 3x faster than MySQL for my usecase. The app is
> written in Java so I would be forced to use JDBC to talk to MySQL. The
> wrapping/unwrapping overhead is a killer.

I've seen a java library that solves this exact problem - I can't remember 
what it was called, but it's page talked about massive speedups because it 
could somehow avoid the insane wrap-unwrap for each little object. Maybe it 
could help.

-- 
Regards,
Christian Iversen

Re: 15M files

[PFC]

Isn't this the RAM-based Prevalence ?

On Sat, 20 Aug 2005 11:58:59 +0200, Christian Iversen  
<chrivers@iversen-net.dk> wrote:

> On Saturday 20 August 2005 00:44, studdugie wrote:
>> MySQL is too slow for the task, even over localhost. I've benchmarked
>> it. Berkeley DB was 3x faster than MySQL for my usecase. The app is
>> written in Java so I would be forced to use JDBC to talk to MySQL. The
>> wrapping/unwrapping overhead is a killer.
>
> I've seen a java library that solves this exact problem - I can't  
> remember
> what it was called, but it's page talked about massive speedups because  
> it
> could somehow avoid the insane wrap-unwrap for each little object. Maybe  
> it
> could help.
>

Re: 15M files

[Christian Iversen]

On Saturday 20 August 2005 12:23, PFC wrote:
> Isn't this the RAM-based Prevalence ?

Yes, I believe that was it. 

-- 
Regards,
Christian Iversen

Re: 15M files

[Hans Reiser]

studdugie wrote:

>Hello. I'm looking to replace a couple Berkeley DB data stores w/
>regular file system directories backed by reiserfs (3.6). The reason
>is Berkeley DB is slow especially for data w/ little or no locality of
>reference. I'm posting to this list because I would like to get some
>opinions on if reiserfs is suitable for the job. Currently there are
>15,079,597 records in 1 of the database. If I moved to a directory
>based db it would result in 15,079,597 discreet files ranging in sizes
>from 1 byte to 1kb. I was reading the FAQ on the namesys site and it
>mentioned that the r5 hash supports 1,200,000 files w/o collision.
>Since 15M is 12.5x greater I'm expecting massive amounts of
>collisions. So the question becomes how bad should I expect it to be?
>Should I assume the file system can handle it or slow to a crawl?  I
>would really appreciate some feedback from the experts before I go
>ripping out the Berkeley DB code.
>
>Thanx.
>
>
>  
>
Use V4, it has much better hashiing.

Re: 15M files

[studdugie]

I can't use V4 because I can't introduce an unstable kernel on the box
where the app is running.

On 8/19/05, Hans Reiser <reiser@namesys.com> wrote:
> studdugie wrote:
> 
> >Hello. I'm looking to replace a couple Berkeley DB data stores w/
> >regular file system directories backed by reiserfs (3.6). The reason
> >is Berkeley DB is slow especially for data w/ little or no locality of
> >reference. I'm posting to this list because I would like to get some
> >opinions on if reiserfs is suitable for the job. Currently there are
> >15,079,597 records in 1 of the database. If I moved to a directory
> >based db it would result in 15,079,597 discreet files ranging in sizes
> >from 1 byte to 1kb. I was reading the FAQ on the namesys site and it
> >mentioned that the r5 hash supports 1,200,000 files w/o collision.
> >Since 15M is 12.5x greater I'm expecting massive amounts of
> >collisions. So the question becomes how bad should I expect it to be?
> >Should I assume the file system can handle it or slow to a crawl?  I
> >would really appreciate some feedback from the experts before I go
> >ripping out the Berkeley DB code.
> >
> >Thanx.
> >
> >
> >
> >
> Use V4, it has much better hashiing.
>

Re: 15M files

[David Masover]

studdugie wrote:
> I can't use V4 because I can't introduce an unstable kernel on the box
> where the app is running.

Search the archives.  I wrote a sort of mini-howto on how to set up the
MM-kernel version of Reiser4 on a stable 2.6 kernel.  You don't even
need my patch if you do 2.6.13-rc6 with the latest MM patch (that you're
grabbing reiser4-specific stuff from)

That's if you trust Namesys that it's stable.  It's reasonably stable,
but maybe not mission-critical-stable yet, in that it hasn't been
hammered by millions of users yet.  But, it works for me.

Re: 15M files

[Hans Reiser]

studdugie wrote:

>I can't use V4 because I can't introduce an unstable kernel on the box
>where the app is running.
>  
>
I suggest you ask vs to send you a patch for the stable kernel on
wednesday or so after we send our latest bundle to akpm (probably on
monday we will send it).

>On 8/19/05, Hans Reiser <reiser@namesys.com> wrote:
>  
>
>>studdugie wrote:
>>
>>    
>>
>>>Hello. I'm looking to replace a couple Berkeley DB data stores w/
>>>regular file system directories backed by reiserfs (3.6). The reason
>>>is Berkeley DB is slow especially for data w/ little or no locality of
>>>reference. I'm posting to this list because I would like to get some
>>>opinions on if reiserfs is suitable for the job. Currently there are
>>>15,079,597 records in 1 of the database. If I moved to a directory
>>>based db it would result in 15,079,597 discreet files ranging in sizes
>>>      
>>>
>>>from 1 byte to 1kb. I was reading the FAQ on the namesys site and it
>>    
>>
>>>mentioned that the r5 hash supports 1,200,000 files w/o collision.
>>>Since 15M is 12.5x greater I'm expecting massive amounts of
>>>collisions. So the question becomes how bad should I expect it to be?
>>>Should I assume the file system can handle it or slow to a crawl?  I
>>>would really appreciate some feedback from the experts before I go
>>>ripping out the Berkeley DB code.
>>>
>>>Thanx.
>>>
>>>
>>>
>>>
>>>      
>>>
>>Use V4, it has much better hashiing.
>>
>>    
>>
>
>
>  
>

Re: 15M files

[Lexington Luthor]

studdugie wrote:
> Hello. I'm looking to replace a couple Berkeley DB data stores w/
> regular file system directories backed by reiserfs (3.6). The reason
> is Berkeley DB is slow especially for data w/ little or no locality of
> reference. I'm posting to this list because I would like to get some
> opinions on if reiserfs is suitable for the job. Currently there are
> 15,079,597 records in 1 of the database. If I moved to a directory
> based db it would result in 15,079,597 discreet files ranging in sizes
> from 1 byte to 1kb. I was reading the FAQ on the namesys site and it
> mentioned that the r5 hash supports 1,200,000 files w/o collision.
> Since 15M is 12.5x greater I'm expecting massive amounts of
> collisions. So the question becomes how bad should I expect it to be?
> Should I assume the file system can handle it or slow to a crawl?  I
> would really appreciate some feedback from the experts before I go
> ripping out the Berkeley DB code.
> 
> Thanx.
> 

Use the SkipDB library - I have a similar code in one of my programs, 
and SkipDB is almost 3x the speed of BDB (plus you don't lose control of 
transaction safety).

LL